Skip to content

ObjectForesight: Predicting 3D Object Trajectories from Human Videos

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://objectforesight.github.io
Area: 3D Vision
Keywords: 3D future prediction, object-centric dynamics, 6-DoF trajectory, egocentric video, diffusion model

TL;DR

ObjectForesight is an object-centric 3D forward dynamics model that predicts physically consistent future 6-DoF object trajectories from passive egocentric videos, powered by an automated curation pipeline yielding 2+ million clean trajectories and a geometry-conditioned Diffusion Transformer.

Background & Motivation

Humans possess an intuitive ability to imagine physical interactions and anticipate how objects move when manipulated—mentally simulating a mug being lifted, a knife slicing food, or a container lid being shut. This forward modeling capability is fundamental to embodied intelligence, enabling humans to plan, predict, and interact effectively in complex physical environments. However, computational models striving for world prediction have predominantly operated in 2D pixel space or abstract latent representations, frequently suffering from spatial drift, non-physical deformations, and a lack of explicit geometric grounding.

The core tension stems from two major obstacles: on the one hand, robot manipulation datasets are scarce, narrow in domain, and reliant on low-level action annotations; on the other hand, massive collections of natural human egocentric video provide rich interaction diversity but completely lack 3D annotations, metric depth, and ground-truth camera poses. Furthermore, object motion induced by human actions exhibits intrinsic multi-modality—a grasped object can be translated, tilted, or placed down along diverse plausible trajectories. Standard autoregressive models regression-trained with single-mode objectives inevitably collapse to the mean trajectory, leading to severe compounding errors and unnatural motion damping.

To circumvent the dependency on specialized robotic hardware and manual control labels, this paper frames passive first-person human video as a direct supervisory source for learning 3D physical dynamics. Core idea: build an automated multi-stage curation pipeline to distill millions of metric-grounded 3D trajectories from in-the-wild egocentric videos, and train a geometry-aware Diffusion Transformer (DiT) operating on depth-normalized SE(3) pose tokens to capture multimodal 6-DoF object motion distributions.

Method

Overall Architecture

The ObjectForesight framework comprises two integrated components: an automated large-scale data curation pipeline that turns uncalibrated egocentric human videos into clean, metric-scaled, anchor-frame-canonicalized 6-DoF trajectories; and a predictive neural dynamics model combining a 3D geometry encoder with an AdaLN-Zero-conditioned Diffusion Transformer (DiT).

Given an observation of \(C\) history frames (default \(C=3\)) with corresponding images, object masks, and 9D pose tokens \(\mathbf{P}_{1:C}\), alongside an anchor-frame point cloud \(\mathbf{X}\) backprojected from monocular depth at frame \(t_a = C+1\), the objective is to predict the distribution over the next \(H\) future 6-DoF object poses \(\mathbf{P}_{\text{future}}\) (default \(H=8\)). All trajectories are canonicalized relative to the anchor-frame camera coordinate frame, effectively isolating genuine object dynamics from observer head motion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Input: Anchor Point Cloud X<br/>+ Context Poses & Bounding Boxes"] --> B["Multimodal Spatiotemporal Context Encoding<br/>Anchor temporal attention & FiLM conditioning"]
    B --> C["Object-Centric Geometric Encoding<br/>PTV3 dual-coordinate point cloud feature pooling"]
    C --> D["Depth-Normalized Pose Tokenization<br/>Log-depth and perspective projection transform"]
    D --> E["Diffusion Temporal Denoising<br/>DiT v-parameterization trajectory generation"]
    E --> F["SE(3) Pose Decoding & Physical Smoothing<br/>Output future H-step 6-DoF trajectory"]

Key Designs

1. Automated 3D Trajectory Curation Pipeline: Mining Metric Supervision from Unconstrained Videos

To extract accurate 3D trajectories from raw, noisy egocentric video without manual intervention, the authors develop an eight-stage automated pipeline. Action segments from EPIC-Kitchens (\(\le 10\) seconds) with detected hands via EgoHOS are first selected. SAM2 is then initialized with positive prompts inside the object mask and negative prompts from hands and neighboring margins, reinforced by a temporal consensus window to suppress flickering. Next, a two-stage filter using the InternVL3 vision-language model discards static objects at the clip level and severely occluded or blurry instances at the frame level. For surviving sequences, TRELLIS reconstructs a clean 3D object mesh template, while SpaTrackerV2 infers metric depth and camera poses. Finally, FoundationPose tracks 6-DoF object poses with robust metric scale estimation, multi-view initialization, and bi-directional tracking with local re-registration whenever projection IoU drops below 0.1, yielding sliding-window trajectories canonicalized to the anchor camera frame.

2. Object-Centric Geometry-Aware Scene Encoding: Fusing Structure with Recent Motion Context

Future physical trajectory depends heavily on local scene boundaries (e.g., table planes, supporting surfaces) and immediate motion momentum. The model projects the concatenated 9D pose and normalized 2D bounding boxes of conditioning frames into a 64D space, applying anchor-queried temporal attention with sinusoidal relative time embeddings to produce a single context vector \(\mathbf{ctx} \in \mathbb{R}^{64}\). Concurrently, the anchor depth point cloud \(\mathbf{X}\) is ingested by a PointTransformerV3 (PTV3) backbone, where each point is explicitly parameterized in both anchor-camera and estimated object-local coordinates. Features are modulated by \(\mathbf{ctx}\) via Feature-wise Linear Modulation (FiLM), followed by an object-centric attention pooling head that biases weights toward points near the target object to construct a compact scene embedding \(\mathbf{z}_{\text{geom}} \in \mathbb{R}^{512}\).

3. Depth-Normalized Pose Tokenization & Prefix-Conditioned Diffusion Transformer: Capturing Multimodal Trajectories

Directly denoising absolute Cartesian translations introduces optimization instability due to high dynamic variance across varying depths. The framework reparameterizes each 9D pose token \(\mathbf{p}_t = [x_t, y_t, z_t, \mathbf{r}_{t,\text{6D}}]\) into depth-normalized coordinates \(\mathbf{y}_t = [u_t, v_t, s_t, \mathbf{r}_{t,\text{6D}}]\): $\(u_t = \frac{x_t}{z_t}, \quad v_t = \frac{y_t}{z_t}, \quad s_t = \log z_t\)$ followed by channel-wise standardization over precomputed dataset statistics. A Diffusion Transformer backbone processes the noised sequence with AdaLN-Zero blocks, conditioning on timestep embeddings, the global geometric vector \(\mathbf{z}_{\text{geom}}\), and a prepended prefix of clean historical pose tokens. Operating under a cosine noise schedule and \(v\)-parameterization with P2 loss weighting and horizon-dependent loss scaling, the model accurately captures the diverse, one-to-many distribution of human-induced object displacements while suppressing trajectory jitter.

Loss & Training

The overall training objective combines denoising loss in normalized token space with physical SE(3) trajectory regularization: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{v}} + \mathcal{L}_{\text{aux}} + \mathcal{L}_{z_{\min}} + \lambda_{\text{vel}} \mathcal{L}_{\text{vel}} + \lambda_{\text{acc}} \mathcal{L}_{\text{acc}}\)$ The primary objective \(\mathcal{L}_{\text{v}}\) is an SNR-weighted mean squared error under \(v\)-parameterization: $\(\mathcal{L}_{\text{v}} = \mathbb{E} \left[ w(\tau) \|\mathbf{v}_\theta(\tilde{\mathbf{Y}}_\tau, \tau) - \mathbf{v}_\tau\|_2^2 \right]\)$ The auxiliary pose loss \(\mathcal{L}_{\text{aux}}\) evaluates decoded physical translations and SO(3) rotations against ground truth via geodesic distance \(d_{\text{geo}}(\mathbf{R}_k, \hat{\mathbf{R}}_k)\), downweighted by the signal coefficient \(\bar{\alpha}_\tau\) on high-noise steps. Smoothness is enforced via \(\mathcal{L}_{\text{vel}}\) and \(\mathcal{L}_{\text{acc}}\) penalizing first and second differences of translation and rotation increments. Degenerate predictions penetrating the camera plane are penalized by a depth-floor loss \(\mathcal{L}_{z_{\min}}\). Weights are set to \(\lambda_R = 2.0, \lambda_{\text{trans}} = 20.0, \lambda_{\text{vel}} = 0.5\), and \(\lambda_{\text{acc}} = 0.1\). At inference time, trajectories are generated using 50-step deterministic DDIM sampling.

Key Experimental Results

Main Results

ObjectForesight is benchmarked on both the in-the-wild curated EPIC-Kitchens trajectory dataset and the clean, lab-collected HOT3D-Clips dataset across translation metrics (ADE, FDE, DES in meters) and rotation metrics (ARE, FRE, RES in degrees). On a 20-video validation subset of EPIC-Kitchens, it is also compared against state-of-the-art video generation (Luma AI Ray3) coupled with the same pose estimation pipeline.

The results from Table 1 of the paper are presented below:

Dataset / Setting Method ADE (m) ↓ FDE (m) ↓ DES (m) ↓ ARE (°) ↓ FRE (°) ↓ RES (°) ↓
Epic-Kitchens Constant Velocity 0.027 0.053 0.007 2.47 5.60 0.80
ObjectForesight-AR 0.067 0.074 0.002 9.48 12.58 0.93
ObjectForesight-DiT (Ours) 0.016 0.029 0.004 2.30 4.82 0.66
vs. Video Gen (Subset) Luma AI Ray3 0.084 0.149 0.020 12.86 20.90 2.62
ObjectForesight-DiT (Ours) 0.029 0.059 0.008 7.29 13.98 1.77
HOT3D-Clips Constant Velocity 0.136 0.280 0.040 38.70 68.53 9.85
ObjectForesight-AR 0.055 0.082 0.007 9.80 14.95 1.55
ObjectForesight-DiT (Ours) 0.021 0.026 0.003 8.92 12.58 1.16

Ablation Study

Ablations on EPIC-Kitchens investigate the choice of geometric 3D scene encoder and DiT parameter scaling.

The results from Table 2 of the paper are summarized below:

(a) Scene Encoders (Fixed 12L-768D DiT) ADE (m) ↓ FDE (m) ↓ ARE (°) ↓ FRE (°) ↓ (b) DiT Scaling (Fixed PTV3) ADE (m) ↓ FDE (m) ↓ ARE (°) ↓ FRE (°) ↓
DGCNN 0.0171 0.0297 2.333 4.968 6L-384D 0.0193 0.0311 4.242 7.762
PointNet++ 0.0171 0.0298 2.357 5.024 8L-512D 0.0171 0.0294 2.802 5.663
SparseConv 0.0179 0.0295 2.700 5.299 12L-768D (Full Model) 0.0165 0.0287 2.299 4.816
No-Encoder 0.0174 0.0298 2.690 5.380
PTV3 (Ours) 0.0165 0.0287 2.299 4.816

Key Findings

  • Diffusion outperforms Autoregressive modeling: On EPIC-Kitchens, the deterministic autoregressive baseline ObjectForesight-AR falls behind even Constant Velocity extrapolation (ADE 0.067 m vs 0.027 m), because mode-averaging across diverse actions degrades trajectory precision. In contrast, the DiT model cuts ADE by 41% (0.016 m) and FDE by 45% (0.029 m).
  • Direct SE(3) modeling surpasses 2D video generation: Comparing against Luma Ray3 demonstrates that inferring motion from generated pixels suffers heavily from visual artifacts and tracking drift (ADE 0.084 m, ARE 12.86°), whereas direct 3D trajectory forecasting yields vastly superior geometric fidelity (ADE 0.029 m, ARE 7.29°).
  • Geometric backbone quality is critical: A weak encoder like SparseConv performs slightly worse than using No-Encoder (ARE 2.700° vs 2.690°), proving that noisy geometric representations compromise dynamics prediction. Conversely, PTV3 with dual-coordinate point features achieves the lowest errors across all metrics.

Highlights & Insights

  • Scalable automated 3D pseudo-ground-truth curation: Orchestrating EgoHOS, SAM2 temporal consensus, InternVL3 gating, TRELLIS mesh recovery, and FoundationPose with metric depth alignment enables mining 2+ million clean 3D trajectories from unconstrained human video without manual labeling.
  • Depth-normalized 9D pose tokenization: Decoupling 3D translation into normalized image coordinates and logarithmic depth effectively prevents numerical instability in the diffusion objective caused by wide variations in camera distance.
  • Anchor-canonicalized dynamics: Re-projecting all trajectory steps into the anchor frame's camera coordinate system eliminates observer ego-motion, isolating true object dynamics and affordance patterns.

Limitations & Future Work

  • Rigid-body assumption: The pipeline currently relies on rigid mesh templates and 6-DoF tracking, precluding non-rigid, fluid, or articulated objects commonly encountered in daily life (e.g., dough, folding clothes, scissors).
  • Short forecast horizon: Evaluations focus on short horizons (\(H=8\), approximately 0.13s to 1.3s), leaving long-horizon multi-stage manipulation forecasting to future exploration.
  • Absence of closed-loop robotic actuation: The model outputs object trajectory rather than robot action commands; integrating predicted 3D object motion as an affordance guide into downstream imitation or reinforcement learning policies remains an open direction.
  • vs. Video World Models (e.g., SVD, Luma Ray3): While video models synthesize appearance changes in pixel space, they lack 3D physical constraints and object permanence; ObjectForesight operates directly in SE(3) space, guaranteeing geometric consistency.
  • vs. Robot Policy Learning (e.g., RT-X, Open X-Embodiment): Robot datasets require explicit low-level action logs and platform-specific teleoperation; ObjectForesight extracts physics and affordance priors directly from internet-scale human video without action supervision.
  • vs. Egocentric Hand / Contact Forecasters (e.g., HandsonVLM, ContactGrasp): Prior egocentric methods forecast hand trajectories or contact hotspots; this work shifts the focus to object-centric dynamics, modeling the direct consequence of human manipulation on physical items.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formalizes 3D object dynamics forecasting from passive human video and introduces an automated multi-stage pipeline yielding a 2M-trajectory dataset.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across two benchmarks, compared against autoregressive, extrapolation, and state-of-the-art video generation baselines, with detailed ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clearly structured, transparent about assumptions, with detailed pipeline descriptions and formal notation.
  • Value: ⭐⭐⭐⭐⭐ Provides a foundational data-driven blueprint for learning physically grounded 3D object dynamics from vast observational video data for embodied AI.