EgoExoMoCap: Distributed Human Motion Capture via Ego- and Exocentric Body Tracking from Head-Mounted Devices¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://siplab.org/projects/EgoExoMoCap
Area: Human Understanding (human_understanding)
Keywords: Distributed MoCap, Egocentric Vision, Exocentric Tracking, Ray-based Pose Representation, Visibility Gating
TL;DR¶
Addressing the reliance of traditional motion capture on bulky multi-camera rigs or obtrusive suits and the failure of single egocentric wearable devices to faithfully reconstruct lower-body motions, EgoExoMoCap introduces a distributed mocap framework requiring only two or more people wearing head-mounted devices (such as Project Aria glasses); it combines continuous egocentric tracking with intermittent exocentric canonicalized 3D rays and DINOv3-based learned visibility gating to achieve robust, high-fidelity 3D global motion reconstruction even under severe occlusions and out-of-view movements.
Background & Motivation¶
Accurate and robust human motion capture in unconstrained real-world environments is fundamental for embodied AI, collaborative VR/AR, and interactive humanoid agents. However, traditional motion capture setups rely heavily on specialized multi-camera studio infrastructure or cumbersome motion capture suits, making them costly and physically restrictive. Recently, the proliferation of lightweight head-mounted displays (HMDs, such as Project Aria glasses) equipped with cameras and visual-inertial SLAM has popularized egocentric motion capture. Nevertheless, purely egocentric tracking suffers from an inherent physical blind spot: the wearer's torso and lower limbs are frequently out of view of the downward/egocentric cameras. Consequently, relying solely on head (and sparse wrist) tracking trajectories often synthesizes statistically plausible motions but fails to faithfully recover true lower-body articulations, such as sitting, crouching, or kneeling.
Conversely, exocentric human motion capture methods using a single external monocular camera (e.g., a smartphone or third-person observer) struggle with scale and depth ambiguities in world coordinates. They are also easily destabilized by dynamic camera motion, motion blur, and pervasive environmental occlusions. Historically, egocentric and exocentric tracking paradigms have been developed in isolation. This overlooks a natural collaborative opportunity in multi-user settings: each participant wearing an HMD is simultaneously a motion subject and a mobile observer of others.
Unifying these heterogeneous signals across multiple wearers introduces two primary challenges: first, coordinating spatial frames across observers subject to rapid head rotations, and second, coping with the intermittent nature of exocentric observations where wearers frequently leave the field of view or are heavily occluded by furniture during human-scene interactions. The core idea is to use an egocentric coarse pose prior to generate region proposals on exocentric views, lift 2D keypoints to depth-scaled 3D rays canonicalized into the wearer's head frame, gate these rays using DINOv3 global semantic context, and fuse the resulting tokens through spatial and temporal Transformers while analytically solving global root translation via forward kinematics.
Method¶
Overall Architecture¶
The EgoExoMoCap pipeline operates in three main stages: first, an ego-only temporal network (EgoNet) consumes the wearer's continuous tracking streams to coarsely predict body pose and project joint locations onto the observer's frame to form region proposals (RoIs); second, 2D keypoints detected within the RoI are unprojected into rotation-invariant, depth-scaled 3D rays canonicalized into the wearer's head frame, while DINOv3 features extracted from the same crop predict per-joint visibility confidence gates; third, Ego and Exo tokens are merged via a Spatial Transformer with an egocentric inductive bias, followed by a full-window bidirectional Temporal Transformer to predict root orientations and joint rotations, with global root position solved analytically via forward kinematics.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Inputs: Wearer & Observer Head/Wrist Trajectories + Observer Images"] --> B["EgoNet Coarse Estimation & Region Proposals"]
B --> C["Exocentric Ray Geometric Canonicalization"]
B --> D["DINOv3 Global Context Visibility Gating"]
C --> E["Ego-Exo Spatio-Temporal Fusion & Analytical Solving"]
D --> E
E --> F["Outputs: Global Full-Body SMPL Motion Trajectory"]
Key Designs¶
1. EgoNet Coarse Estimation & Region Proposals: Eliminating Reliance on Generic 2D Detectors Under heavy occlusions, dynamic viewpoints, and rapid observer head movements, generic object detectors (e.g., YOLO) often fail or mislocalize the target person. To address this, EgoExoMoCap uses the wearer's own continuous egocentric tracking streams as an anchor. The wearer's head position, orientation, and velocities—augmented by wrist trajectories when available and normalized by a 6D planar displacement relative to the first frame—are encoded into a feature vector \(\mathbf{x}_t^{\text{ego}} \in \mathbb{R}^{60}\). A lightweight temporal network (EgoNet), composed of a sinusoidal temporal embedding and an MLP-Mixer block, mixes features across time and channels to coarsely predict initial global orientations and SMPL body poses. 3D joints derived from this coarse prediction are projected onto the observer's camera image using known calibration parameters, and an axis-aligned bounding box with a fixed margin forms a reliable region of interest (RoI). This guarantees that subsequent exocentric processing consistently tracks the wearer without being misled by nearby background individuals or temporary occlusions.
2. Exocentric Ray Geometric Canonicalization: Decoupling Observer Head Rotations and Scale Ambiguity Inside the extracted RoI, ViTPose detects \(K=13\) 2D body keypoints. Directly passing raw 2D pixel coordinates into the fusion network would entangle observer-specific camera intrinsics and transient head orientations. To overcome this, each 2D keypoint \(\mathbf{x}_{t,j}\) is unprojected into a 3D unit ray in world coordinates using the observer's camera intrinsics \(\mathbf{K}_{\text{obs}}\) and rotation \(\mathbf{R}_{\text{obs}}\), effectively factoring out the high-frequency rotational variance of the observer's head motion: $\(\hat{\mathbf{d}}_{t,j} = \frac{\mathbf{R}_{\text{obs}} \mathbf{K}_{\text{obs}}^{-1} \tilde{\mathbf{x}}_{t,j}}{\|\mathbf{R}_{\text{obs}} \mathbf{K}_{\text{obs}}^{-1} \tilde{\mathbf{x}}_{t,j}\|}\)$ Because the ray lacks absolute metric distance, it is scaled by the Euclidean distance between observer and wearer head positions and anchored at the observer's global head position to yield a proxy 3D endpoint \(\mathbf{e}_{t,j}\): $\(\mathbf{e}_{t,j} = \|\mathbf{p}_t^{w,\text{head}} - \mathbf{p}_t^{o,\text{head}}\| \cdot \hat{\mathbf{d}}_{t,j} + \mathbf{p}_t^{o,\text{head}}\)$ To make the representation independent of the wearer's absolute position and path through the world, each endpoint is transformed into the canonical coordinate frame of the wearer's head: $\(\mathbf{r}_{t,j} = \mathbf{R}_{w,\text{head}}^{-1} (\mathbf{e}_{t,j} - \mathbf{p}_t^{w,\text{head}})\)$ The concatenated features \(\mathbf{r}_t \in \mathbb{R}^{3K}\) establish a canonical geometric representation that is invariant to observer-wearer distance, observer camera rotation, and wearer global trajectory.
3. DINOv3 Global Context Visibility Gating: Suppressing Occlusion and Out-of-View Artifacts During natural human-scene interactions, portions of the wearer's body are frequently obstructed by tables, chairs, or objects, or they drift outside the observer's view. Standard 2D keypoint detectors often output erroneous predictions with spuriously high confidences in these ambiguous regions. To dynamically modulate exocentric input quality, the method extracts a global classification token \(\mathbf{F}_{\text{CLS}} \in \mathbb{R}^{768}\) using a frozen DINOv3 backbone on the RoI crop. A lightweight two-layer MLP (ScoreNet, \(768 \to 512 \to K\)) equipped with a Sigmoid activation predicts per-joint confidence scores: $\(\mathbf{w}_t = \sigma(\text{MLP}(\mathbf{F}_{\text{CLS}})) \in [0, 1]^K\)$ Each canonical ray is element-wise weighted: \(\hat{\mathbf{r}}_{t,j} = w_{t,j} \cdot \mathbf{r}_{t,j}\). Leveraging the holistic scene semantic awareness of DINOv3, the gating mechanism suppresses corrupted keypoint rays when body parts are occluded behind obstacles, allowing the fusion architecture to gracefully fall back on egocentric motion priors.
4. Ego-Exo Spatio-Temporal Fusion & Analytical Solving: Inductive Bias and Drift-Free Kinematics The egocentric signals and EgoNet's coarse pose predictions are concatenated into an Ego Token, while the gated canonical rays are projected into an Exo Token. In each frame, the token pair \([e_t, o_t]\) is fed into a Spatial Transformer Encoder. Crucially, only the transformed Ego Token output \(\tilde{e}_t\) is retained as the frame feature. This asymmetric retention enforces an egocentric inductive bias: the egocentric stream acts as the reliable primary backbone, while the exocentric observation provides auxiliary geometric refinement. This formulation naturally scales to \(N\) observers by expanding the sequence to \([e_t, o_t^1, \dots, o_t^N]\) without modifying network weights. Across a temporal window of \(T=96\) frames, features with learnable temporal embeddings pass through a bidirectional Temporal Transformer Encoder. Lightweight MLP heads decode the root orientation \(\hat{\boldsymbol{\theta}}_t^{w,\text{root}} \in \mathbb{R}^6\) and \(J=21\) joint rotations \(\hat{\boldsymbol{\theta}}_t^{w,\text{body}} \in \mathbb{R}^{126}\). Rather than predicting root translation via error-prone network regression, the global root translation is computed analytically via forward kinematics from the known head position: \(\mathbf{p}_t^{w,\text{root}} = \mathbf{p}_t^{w,\text{head}} - \text{FK}_{\text{head}}(\hat{\boldsymbol{\theta}}_t)\), completely preventing trajectory drift.
Loss & Training¶
The system is trained in two distinct stages: 1. Stage 1 (EgoNet Pretraining): EgoNet is trained purely on egocentric tracking inputs, without exocentric visual data, establishing an independent motion prior. 2. Stage 2 (End-to-End Fusion Training): EgoNet and the DINOv3 visual backbone are frozen. The ray embeddings, ScoreNet, Spatial Transformer, Temporal Transformer, and decoding heads are trained end-to-end using a multi-term \(L_1\) objective: $\(\mathcal{L} = \lambda_{\text{orient}} \mathcal{L}_{\text{orient}} + \lambda_{\text{rot}} \mathcal{L}_{\text{rot}} + \lambda_{\text{pos}} \mathcal{L}_{\text{pos}}\)$ Here, \(\mathcal{L}_{\text{orient}}\) supervises the 6D root orientation, \(\mathcal{L}_{\text{rot}}\) supervises the 6D local rotations across all \(J=21\) body joints, and \(\mathcal{L}_{\text{pos}}\) penalizes Euclidean errors on 3D joint Cartesian positions derived through SMPL forward kinematics. Loss weights are set to \(\lambda_{\text{orient}} = 0.02\), \(\lambda_{\text{rot}} = 1.0\), and \(\lambda_{\text{pos}} = 1.0\).
Key Experimental Results¶
Main Results¶
The method was evaluated on Nymeria (a massive 300-hour indoor/outdoor dataset collected with Project Aria glasses and Xsens ground truth) and EgoHumans (an in-the-wild multi-person dataset capturing dynamic sports like fencing, basketball, and badminton). Experiments cover both 3-point tracking (glasses + 2 wristbands) and 1-point tracking (glasses only). Metrics include Mean Per-Joint Position Error (MPJPE, cm), Upper-body error (Upper, cm), Lower-body error (Lower, cm), Mean Per-Joint Velocity Error (MPJVE, cm/s), and Motion Jitter (\(10^2 \text{ m/s}^3\)).
Table 1: Quantitative Evaluation on Nymeria
| Methods | Modality | 3-pt MPJPE (cm) | 3-pt Upper (cm) | 3-pt Lower (cm) | 3-pt MPJVE (cm/s) | 3-pt Jitter | 1-pt MPJPE (cm) | 1-pt Upper (cm) | 1-pt Lower (cm) | 1-pt MPJVE (cm/s) | 1-pt Jitter |
|---|---|---|---|---|---|---|---|---|---|---|---|
| AvatarPoser [39] | Ego | 8.16 | 3.82 | 14.43 | 13.62 | 2.20 | 12.38 | 9.22 | 16.94 | 18.90 | 4.07 |
| EgoPoser [38] | Ego | 7.74 | 3.44 | 13.96 | 13.94 | 2.43 | 11.86 | 9.08 | 15.87 | 20.26 | 3.43 |
| EgoAllo [99] | Ego | 8.77 | 3.18 | 16.84 | 12.81 | 1.72 | 12.53 | 8.27 | 18.68 | 20.17 | 1.96 |
| RPM [12] | Ego | 12.19 | 3.50 | 24.74 | 17.62 | 1.29 | 16.83 | 9.39 | 27.58 | 18.93 | 1.15 |
| PromptHMR [90] | Exo | 13.48 | 8.88 | 20.13 | 26.78 | 5.17 | 13.48 | 8.88 | 20.13 | 26.78 | 5.17 |
| PromptHMR-Finetuned | Exo | 10.65 | 7.63 | 15.01 | 23.55 | 4.98 | 10.65 | 7.63 | 15.01 | 23.55 | 4.98 |
| PromptHMR+EgoPoser | EgoExo | 6.47 | 3.41 | 10.90 | 17.49 | 3.08 | 9.03 | 6.74 | 12.34 | 17.52 | 3.17 |
| EgoExoMoCap (Ours) | EgoExo | 5.72 | 2.72 | 10.05 | 12.77 | 2.16 | 8.28 | 6.10 | 11.44 | 17.23 | 2.47 |
Table 2: Quantitative Evaluation on EgoHumans (Cross-Dataset & Multi-Observer Setup)
| Methods / Config | Modality | 3-pt MPJPE (cm) | 3-pt Upper (cm) | 3-pt Lower (cm) | 3-pt MPJVE (cm/s) | 3-pt Jitter | 1-pt MPJPE (cm) | 1-pt Upper (cm) | 1-pt Lower (cm) | 1-pt MPJVE (cm/s) | 1-pt Jitter |
|---|---|---|---|---|---|---|---|---|---|---|---|
| AvatarPoser [39] | Ego | 9.60 | 4.34 | 17.18 | 32.94 | 3.08 | 14.17 | 10.38 | 19.64 | 54.60 | 3.84 |
| EgoPoser [38] | Ego | 9.03 | 3.87 | 16.48 | 33.12 | 3.55 | 13.96 | 10.49 | 18.96 | 57.04 | 5.05 |
| PromptHMR-Finetuned | Exo | 18.15 | 11.79 | 27.34 | 46.84 | 5.17 | 18.15 | 11.79 | 27.34 | 46.84 | 5.17 |
| PromptHMR+EgoPoser | EgoExo | 8.62 | 4.24 | 14.94 | 34.01 | 2.79 | 13.27 | 8.96 | 19.50 | 41.88 | 2.71 |
| EgoExoMoCap (Ours) | EgoExo | 7.62 | 3.45 | 13.64 | 30.94 | 1.83 | 11.54 | 8.18 | 16.39 | 41.04 | 2.07 |
| Multi-Observer Subset: | |||||||||||
| EgoExo-single-observer* | EgoExo | 8.80 | 7.00 | 11.40 | 47.23 | 3.59 | 13.70 | 12.35 | 15.64 | 50.32 | 3.01 |
| EgoExo-triangulation* | Ego-Multi-Exo | 8.49 | 6.58 | 11.25 | 44.38 | 3.23 | 13.34 | 11.83 | 15.52 | 49.04 | 2.90 |
| EgoExo-multi-observer (Ours)* | Ego-Multi-Exo | 7.11 | 5.89 | 8.87 | 37.31 | 2.58 | 10.48 | 9.59 | 11.77 | 45.50 | 2.83 |
Ablation Study¶
Ablations on the Nymeria test set dissect the role of input modalities, region proposal mechanisms, learned gating strategies, and geometric coordinate transformations.
Table 3: Ablation Study on Nymeria
| Ablation Category | Variant Config | MPJPE (cm) | Upper (cm) | Lower (cm) | MPJVE (cm/s) | Jitter (\(10^2 \text{ m/s}^3\)) | Key Observation |
|---|---|---|---|---|---|---|---|
| Default | EgoExoMoCap (Full Model) | 5.72 | 2.72 | 10.05 | 12.77 | 2.16 | Full dual-stream fusion model |
| Modality & Proposal | w/o Ego | 11.45 | 8.00 | 16.43 | 34.40 | 6.55 | Severe tracking jitter without continuous anchor |
| w/o Exo | 7.53 | 3.38 | 13.53 | 13.78 | 2.19 | Loss of external visual validation drops lower body accuracy | |
| w/o Ego BBX (using YOLO) | 6.43 | 3.05 | 11.31 | 15.54 | 2.80 | Frequent 2D detector failures under severe occlusion | |
| Learned Gating | w/o DINO Score | 6.30 | 3.00 | 11.08 | 14.42 | 2.38 | Noisy 2D keypoints corrupt downstream Transformer |
| w/ ViT Score | 6.29 | 2.92 | 11.16 | 13.91 | 2.23 | 2D detector confidence is uncalibrated under occlusion | |
| w/ Masking (< 0.2 hard drop) | 6.32 | 2.99 | 11.14 | 15.31 | 2.68 | Hard truncation causes temporal discontinuity | |
| Ray Representation | w/ Ray in World-Space | 6.99 | 3.15 | 12.54 | 13.87 | 2.44 | Vulnerable to observer head rotational motion |
| w/ Ray in Exo-Space | 6.14 | 2.92 | 10.78 | 13.22 | 2.21 | Lacks canonicalization to the wearer's local frame | |
| w/o Depth Scaling | 6.26 | 2.99 | 10.97 | 13.26 | 2.19 | Loss of inter-subject metric distance causes scale drift | |
| w/o Lifting (using raw 2D) | 6.26 | 2.97 | 11.00 | 13.81 | 2.33 | Entangles camera parameters with spatial geometry |
Key Findings¶
- Exocentric Views Faithfully Resolve Lower-Body Ambiguities: Egocentric baselines produce large lower-body errors (13.96 to 27.58 cm). EgoExoMoCap reduces lower-body error to 10.05 cm in the 3-point setup (a ~28% reduction compared to EgoPoser), proving that intermittent third-person observations are critical for resolving occluded limb configurations like sitting and kneeling.
- DINOv3 Context Gating Tames Severe Occlusions: Naive late-fusion baselines (PromptHMR + EgoPoser) produce elevated velocity error and jitter (MPJVE of 17.49 cm/s). DINOv3 semantic gating suppresses unreliable 2D keypoints during furniture occlusions, reducing MPJPE from 6.30 cm to 5.72 cm on Nymeria and from 16.37 cm down to 7.92 cm in complex visual occlusion scenarios.
- Collaborative Multi-Observer Scaling: On the EgoHumans multi-observer subset, moving from a single observer to multiple observers via the Spatial Transformer drops MPJPE from 8.80 cm to 7.11 cm, significantly outperforming classical multi-view geometric triangulation (8.49 cm).
Highlights & Insights¶
- Decentralizing MoCap with Ubiquitous Wearables: Reframes smart glasses from passive, single-user recording sensors into active, collaborative nodes of a distributed motion capture network, removing the need for motion capture suits or dedicated studio camera setups.
- Rotation-Decoupled and Metric-Scaled Ray Canonicalization: By using the observer's camera extrinsics to align 2D keypoints into world directions, scaling by inter-head distance, and canonicalizing into the wearer's local frame, the system eliminates high-frequency observer head jitter and spatial coordinate overfitting.
- High-Level Semantic Vision Guiding Low-Level Geometry: Employs frozen DINOv3 features to learn per-joint visibility confidence weights, allowing holistic scene understanding (e.g., distinguishing body parts from furniture) to filter out corrupted geometric rays.
Limitations & Future Work¶
- Confusion under Multi-Person Inter-Occlusion: When the target wearer is heavily occluded by another moving participant with similar appearance, DINOv3 feature crops may struggle to separate identities, occasionally corrupting visibility weights.
- Pre-Determined Subject Body Shape: The current model assumes SMPL shape identity parameters are known a priori or uses a template mean, without jointly estimating metric body shape from the images or sensor feeds.
- Physical Contact Modeling: In extended periods without exocentric visibility, the model relies on egocentric regression, which can occasionally exhibit minor foot sliding or self-penetration; integrating physics simulation or contact constraints could further enhance realism.
Related Work & Insights¶
- vs EgoPoser [38] / AvatarPoser [39]: Egocentric trackers synthesize plausible postures from sparse head/wrist IMU streams but fail to resolve non-visible lower-body poses; EgoExoMoCap leverages collaborative external viewpoints to provide direct geometric constraints.
- vs PromptHMR [90] / WHAM [73]: Monocular exocentric methods rely on visual SLAM to establish global trajectories, which frequently drifts or breaks under dynamic head movement and blur; EgoExoMoCap grounds the global motion in the wearer's continuous SLAM trajectory, resolving scale and drift ambiguities analytically.
- vs EgoHumans [43] Multi-View Triangulation: Traditional multi-view pipelines require multiple synchronized cameras and struggle when occlusions corrupt individual views; EgoExoMoCap utilizes learned spatial attention to adaptively weigh multiple views according to semantic confidence.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering distributed ego-exo collaborative motion capture using lightweight wearable devices, featuring elegant ray canonicalization and semantic gating.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across two massive real-world benchmarks (Nymeria and EgoHumans) with rigorous ablations and multi-observer analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, intuitive architectural visualizations, and cohesive narrative flow.
- Value: ⭐⭐⭐⭐⭐ Greatly lowers the barrier to gathering in-the-wild, large-scale multi-person 3D interaction data for embodied AI and mixed reality applications.