ECHO: Ego-centric Modeling of Human-Object Interactions

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/ptrvilya/echo
Area: Human Understanding
Keywords: human-object interaction, egocentric tracking, tri-variate diffusion, contact modeling, long-sequence reconstruction

TL;DR

ECHO treats human motion, object trajectories, and contacts as mutually constrained diffusion variables to recover full interactions from head and wrist tracking with known object geometry, combining mixed-data training and smooth inpainting to achieve human MPJPE of 6.8±0.1 cm and object vertex error of 33.5±0.5 cm on BEHAVE.

Background & Motivation

Headsets and wrist-worn devices provide sparse tracking, but three tracked points do not uniquely determine a whole body, let alone the location of an object, whether it is being lifted, or when it contacts the user. A body-only model might generate plausible walking without explaining why the hands remain a particular distance apart. Predicting the body and object separately and then joining them with rules can also produce disconnected hands or floating objects.

Existing egocentric interaction methods often depend on full-body IMUs, scanned scenes, or visual input. Meanwhile, human-object interaction datasets such as BEHAVE and OMOMO have limited scale and motion diversity, whereas AMASS provides extensive human motion without corresponding object trajectories. Requiring complete interaction annotations for every training sample discards a valuable motion prior. Pretraining a body model and attaching a separate interaction component does not, by itself, learn the relationships among human motion, objects, and contacts within one model.

ECHO incorporates missing modalities and sparse observations into generation: observed components can remain clean, unknown components are denoised, and contact explicitly constrains the body-object relationship. Core Idea: make human motion, objects, and contacts condition one another within a single diffusion model, while strengthening the human prior with motion-only data so sparse wearable signals can support continuous interaction reconstruction.

Method

Overall Architecture

Inputs are three-point head and wrist tracking, the known canonical object mesh and class, and known body shape parameters. Outputs are human motion, rigid object trajectories, and contact sequences. Thus, using only three-point tracking refers to the sparse dynamic tracking signals; it does not mean the system discovers arbitrary unknown objects without templates.

Per-frame head-centric representation first organizes the human, object, and contact sequences into jointly denoisable variables. Tri-variate conditional diffusion predicts them together. At inference, smooth inpainting with contact guidance blends old and new predictions in overlapping windows, allowing a fixed-window model to process a continuing stream.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Head and wrist tracking<br/>Object mesh and class"] --> B["Per-frame head-centric<br/>representation"]
    B --> C["Tri-variate<br/>conditional diffusion"]
    D["AMASS and interaction data"] --> C
    C --> E["Smooth inpainting<br/>with contact guidance"]
    E --> F["Continuous human motion<br/>Object trajectories and contacts"]

Key Designs

1. Per-frame head-centric representation: avoid redefining interaction coordinates for each window

If every window chooses its own canonical coordinate system, an object's representation changes across windows and long-sequence stitching requires additional alignment. ECHO extends EgoAllo's per-frame canonicalization, anchoring object translation to the canonicalized head position at each frame instead of choosing an origin only at the beginning of a sequence. An arbitrary contiguous segment can consequently be processed without redefining its representation for a new window.

Object motion is represented by rigid rotation and translation in the head-centric frame, with rotations converted to a 6D representation. The object class is one-hot encoded, and PointNext extracts a 1024-dimensional feature from the canonical mesh vertices; both provide global conditioning. The model therefore receives object identity and geometry that can help relate the distance between the hands to object size. It does not reconstruct the object's shape from the tracking signals.

The human representation uses SMPL-X but predicts only the 21×6 body-pose component and assumes known shape parameters. Finger, eye, and jaw poses are not prediction targets here. The root transformation is inferred by aligning the model's head joint with the tracked head. Conditioning also includes canonicalized head and hand orientations, head-to-floor height, and relative head and hand transformations between adjacent frames.

The head increment in the original Eq. (5) clarifies why conditioning contains more than an absolute position:

\[ \Delta T_{\mathrm{head}}^{t-1,t} =\left(T_{\mathrm{world,head}}^{t-1}\right)^{-1} T_{\mathrm{world,head}}^t. \]

This transformation retains the head's movement from the preceding frame to the current one, with corresponding adjacent-frame relations for the hands. Together with orientation and height, it supplies motion evidence for the body and object without requiring the network to memorize a particular world origin.

2. Tri-variate conditional diffusion: constrain missing components with observed components

The variables are human motion \(H\), object trajectory \(O\), and contact sequence \(I\). Contact is not merely an auxiliary quantity computed from the final meshes: shortest distances between sampled human surface points and the object are mapped through a sigmoid to [0, 1], with a threshold and decay parameter controlling the contact neighborhood. Lower-body joint velocity and ground proximity also define human-ground contacts. Continuous contact values can participate in diffusion and teach the network how contact relates to body and object placement.

The denoiser builds on DiT with rotary positional embeddings. It receives the three noisy sequences, their denoising steps, and tracking and object conditions, predicting clean samples rather than noise. Noise levels can be specified separately: observed human poses can have zero noise while objects and contacts are denoised, and a few observed object frames can constrain their trajectory. The central benefit of independent schedules is freedom to combine conditions, not simply an increase in network branches.

The model's capability must be distinguished from its training procedure. Section 3.2 explicitly samples from \(2^3=8\) combinations of modalities being noisy or provided as clean conditions, but synchronizes the noise level across all modalities that are being diffused for training stability. It would therefore be inaccurate to say that the actual training procedure independently samples three arbitrary noise levels for every example.

For AMASS samples containing only human motion, learnable tokens replace object conditioning and signal that object interactions should be ignored. These samples still contribute a human motion prior, while BEHAVE and OMOMO provide joint interaction supervision. The main paper does not fully specify missing-modality loss masking, so object supervision for AMASS must not be invented. Training also randomly drops hand and object conditions, exposing the model to intermittent observations before evaluation.

3. Smooth inpainting with contact guidance: let adjacent windows revise their overlap

Independent window prediction can cause discontinuities. Ordinary inpainting conditions on the preceding window but discards the new window's estimates in the overlapping region, leaving the historical result in control. At every diffusion step, ECHO instead replaces the overlap with a weighted average of historical and current predictions, gradually transitioning between them rather than switching abruptly at a boundary.

Blending occurs during denoising, not as a single smoothing operation after generating the complete sequence. The overlap length also controls the relationship between historical context and latency: a larger overlap leaves fewer new frames per window. The main paper does not specify the exact weighting function, so a linear blend schedule or a particular window function should not be assumed.

Optional inference guidance enforces contact consistency by relating predicted human and object meshes to the predicted contact vector and adding a foot-floor contact term. Predictions are updated at each denoising step using these geometric constraints. This relies on estimated contacts rather than ground-truth contact labels at inference, but it is still geometric guidance, not a complete guarantee involving mass, friction, and mechanical stability.

A Worked Example

Consider an illustrative sequence of a person carrying an object with both hands. The head trajectory supplies overall body-motion cues, wrist trajectories constrain the arms and grasp locations, and the known object mesh supplies dimensions and shape. Human motion, objects, and contacts are denoised together instead of fixing the body first and subsequently forcing the object near the hands.

At the experimental rate of 30 fps, a window contains 60 frames and adjacent windows overlap by 30 frames, advancing by 30 new frames each time. In the next window, historical and current estimates are progressively blended over the overlap. If reliable tracking is additionally available for some object frames, these can serve as clean conditions to reduce trajectory uncertainty. This example explains the information flow; it is not a separately reported test case.

Loss & Training

The main paper lists six weighted loss terms: reconstruction of the three diffused modalities, object-trajectory smoothness, human joint error, and a foot-skating penalty. Exact weights and full formulas are deferred to supplementary material. The available cache contains the main paper and references, with some equations corrupted by text extraction, so these hyperparameters and the exact contact equation are not reconstructed here.

The model has 57.7M parameters and trains with AdamW, learning rate 5e-4, and batch size 256 for 300k steps on a single RTX 5090, taking approximately 30 hours. Inference uses 100 DDPM steps. A 60-frame window with 30-frame overlap takes 640 ms without guidance or 980 ms with guidance, corresponding to approximately 46 FPS and 30 FPS of new-frame throughput. Window latency is not equivalent to instantaneous per-frame response.

Key Experimental Results

Main Results

Training uses the union of BEHAVE, OMOMO, and AMASS. The first two use official splits, while AMASS follows the EgoAllo split. BEHAVE's SMPL+H annotations are converted to SMPL-X, and all sequences use 30 fps. Training samples 60-frame windows; evaluation performs continuous inference over complete sequences.

The baselines are author-extended versions: BoDiffusion+O adds object prediction, and EgoAllo+H+O adds hand conditioning and object prediction. They share the training-data union and head-centric object coordinates with ECHO. The following results are selected from original Table 1. MPJPE measures mean per-joint human position error, \(E_{v2v}\) measures object vertex-to-vertex error, and \(E_c\) measures object center error; all are in cm and lower is better.

Dataset	Method	MPJPE	\(E_{v2v}\)	\(E_c\)
BEHAVE	BoDiffusion+O	8.3±0.2	44.2±1.2	29.9±1.2
BEHAVE	EgoAllo+H+O	7.6±0.1	39.1±1.1	22.5±0.7
BEHAVE	ECHO	6.8±0.1	33.5±0.5	20.1±0.3
OMOMO	BoDiffusion+O	7.6±0.4	33.2±1.9	22.2±1.7
OMOMO	EgoAllo+H+O	6.6±0.1	30.8±0.9	18.3±0.5
OMOMO	ECHO	6.0±0.1	26.5±1.1	15.2±0.3

The original ± notation is retained. The authors describe the statistics as mean and variance over three runs; this note does not relabel the uncertainty as standard deviation or a confidence interval. Relative to EgoAllo+H+O, ECHO reduces object vertex error by 5.6 cm on BEHAVE and 4.3 cm on OMOMO. However, its foot-contact score in the original table is not best on every dataset, so superiority should not be claimed across all metrics.

Ablation Study

These BEHAVE ablations come from original Table 5. All three errors are in cm, and other training settings are held constant.

Config	MPJPE	\(E_{v2v}\)	\(E_c\)
ECHO	6.8±0.1	33.5±0.5	20.1±0.3
NoGuide	6.8±0.1	33.6±0.5	20.3±0.4
Inpaint w/o smooth	6.9±0.1	33.7±0.5	20.4±0.3
Only \((H,O)\), no contact modality	8.1±0.1	34.4±0.3	20.7±0.1
NoAMASS	8.7±0.1	34.7±0.2	21.8±0.2

Key Findings

• Contact modeling and data scale affect average position errors more than inference refinements in Table 5: removing contacts increases human MPJPE from 6.8 to 8.1, and removing AMASS increases it to 8.7. The smaller changes from guidance and smooth inpainting should not be overstated.
• AMASS evaluation in Table 2 reinforces the importance of the motion prior: full-model MPJPE is 7.4±0.1 cm versus 43.1±0.1 cm for NoAMASS. This also reflects different training-data coverage, not merely a network-architecture advantage.
• Table 3 randomly drops hand tracking while keeping head tracking intact: BEHAVE MPJPE is 7.0±0.2 cm with 50% missing and 9.3±0.5 cm with 90% missing. Random missing observations do not establish robustness to every sensor-noise pattern or prolonged occlusion.
• Table 4 adds 50% observed object frames alongside three-point conditioning: \(E_{v2v}\) falls from 33.46±0.50 to 10.75±0.64 cm, while human MPJPE changes only from 6.82±0.08 to 6.79±0.09 cm. Most improvement remains concentrated in the observed modality.

Highlights & Insights

• Contact is both a generated variable and an inference-time geometric constraint. Unlike detecting collisions only after prediction, this lets interaction relationships influence the full process and supports sparse contacts as additional conditions.
• One network accommodates human-only motion and complete interaction data. AMASS supplies broader motion priors while interaction examples supervise object-contact relationships, offering a practical alternative to relying exclusively on a small HOI dataset.
• Smooth inpainting places window reconciliation inside diffusion. The reusable idea is not another low-pass filter, but allowing newly available context to revise predictions in the historical overlap.

Limitations & Future Work

• The authors explicitly limit environmental contact modeling mainly to the ground, leaving complex dynamic surroundings insufficiently constrained. Arbitrary-length inference does not guarantee long-term physical consistency.
• Missing fine-grained finger tracking limits dexterous interactions with small objects such as pens and scissors. Egocentric RGB or additional hand signals are proposed directions, not visual conditions already used by the current method.
• Known canonical object geometry, class, and body shape remain input assumptions. Object position and orientation are still ambiguous under sparse sensing, so the results should not be interpreted as template-free, precise capture of arbitrary objects.
• Real-world generalization includes the Aria Digital Twin sequence in Fig. 6, but the main paper does not provide large-scale quantitative wearable-deployment evaluation. RTX 5090 window timings cannot be directly extrapolated to phones or headsets.

Related Work & Insights

• vs EgoAllo / BoDiffusion: The original methods focus on human motion, and the evaluated versions add hand or object support. ECHO's distinction is explicit contact and joint multimodal conditioning, not simply a larger output vector.
• vs TriDi: TriDi already models the joint human-object-interaction distribution but operates on static poses and HOI-only data. ECHO applies the idea to sequences and incorporates motion-only data into joint training.
• vs HMD2: Both use historical windows as conditions. ECHO additionally blends historical and current overlap predictions at each diffusion step rather than discarding the new estimates there.
• vs iReplica / IMU-HOI: The former depends on denser body tracking, scene information, and initialization; the latter requires body and object-mounted IMUs. ECHO reduces dynamic tracking requirements while retaining known object geometry.

Rating

• Novelty: 4/5. Combines tri-variate interaction diffusion, mixed data, and continuous inference for sparse egocentric reconstruction, while building on TriDi's distribution-modeling idea.
• Experimental Thoroughness: 4/5. Includes multiple datasets, missing conditions, sparse additional observations, and component ablations, but real deployment evaluation remains limited.
• Writing Quality: 4/5. Clearly relates representation, training, and inference; independent schedule capability must be distinguished from synchronized training noise.
• Value: 4/5. Offers a reusable approach to full-body interaction reconstruction from wearables, subject to object templates, finger detail, and computational latency.