Towards Real-World Wearable Motion Reconstruction

Conference: ECCV 2026
Paper: ECCV Official
Area: Human Understanding
Keywords: Wearable Motion Capture, Multimodal Learning, Flow Matching, Sensor Complementarity, Sparse Pose Estimation

TL;DR

Addressing the unpredictable and heterogeneous combinations of everyday wearable devices, this paper introduces a multimodal benchmark synchronizing commodity smartphones, smartwatches, pressure-sensing insoles, and a VR headset with high-precision 120-camera markerless MoCap, and proposes WHIP, a conditional flow-matching generative model that achieves robust, physically plausible full-body motion reconstruction from arbitrary sensor subsets alongside systematic quantification of sensor complementarity.

Background & Motivation

Human Motion Capture (MoCap) underpins a wide spectrum of applications across virtual/augmented reality, health monitoring, robotics, sports science, and digital animation. However, traditional optical marker-based systems depend on cumbersome sensor suits and tightly controlled capture volumes, while multi-camera markerless capture systems require elaborate external camera arrays, restricting high-fidelity tracking to laboratory environments. This practical bottleneck has spurred rapid progress in lightweight on-body sensing: inertial measurement units (IMUs) have enabled sparse pose estimation with as few as six sensors, yet even these streamlined configurations demand careful strapping and intrusive calibration; head-mounted devices (HMDs) provide reliable upper-body tracking via 6-DoF SLAM, but lower-body estimation remains fundamentally ill-posed; and plantar pressure-sensing insoles supply invaluable contact dynamics and gait signals, yet have largely been evaluated in isolation on narrow locomotion benchmarks.

The core tension in current wearable motion capture lies in the mismatch between algorithmic assumptions and real-world behavior: existing frameworks remain rigidly specialized to fixed hardware topologies (e.g., dedicated 6-IMU suits or headset-plus-controller rigs), failing catastrophically when sensors are missing. In daily life, however, users wear heterogeneous and fluid combinations of commodity devices—carrying a smartphone in a pocket, wearing a smartwatch on one wrist, putting on smart insoles during physical exercise, or donning an HMD indoors. Current paradigms lack both an aligned multimodal dataset spanning these commodity devices and a flexible architectural foundation capable of adapting to arbitrary, dynamic sensor configurations.

To overcome this hardware-bound limitation, this paper targets unobtrusive everyday consumer wearables across both benchmark curation and adaptive generative modeling. Core idea: collect the first large-scale multimodal dataset synchronizing everyday consumer wearables (smartphones, smartwatches, smart insoles, and an HMD) with markerless 120-camera MoCap, and develop WHIP, a conditional flow-matching generative model with modality-dedicated cross-attention that enables plug-and-play full-body motion reconstruction from arbitrary sensor subsets while systematically quantifying sensor complementarity.

Method

Overall Architecture

WHIP reconstructs a temporal sequence of full-body 3D skeletal joint positions \(X = \{J_t\}_{t=1}^T \in \mathbb{R}^{T \times J \times 3}\) from conditioning inputs \(C = \{W, H, I, P, A\}_{\mathcal{M}}\), where \(\mathcal{M}\) represents an arbitrary active subset of smartphones (\(P\)), smartwatches (\(W\)), smart insoles (\(I\)), headset pose (\(H\)), and optional action labels (\(A\)). The model addresses heterogeneous sampling rates, varying local coordinate systems, dynamic sensor dropout, and the inherent one-to-many kinematic ambiguity of sparse observations.

The complete reconstruction pipeline unfolds in three cohesive stages: first, incoming wearable time series are resampled to a unified 30 Hz stream and processed in local device frames; second, sequence-level conditioning variables (flow time \(\tau\) and action label \(a\)) modulate the transformer blocks via adaptive layer normalization (adaLN), while modality-specific MLPs project temporal sensor signals into localized token embeddings mapped into reserved token channels; third, an 8-block Diffusion Transformer (DiT) leverages conditionally evaluated modality-specific cross-attention modules to guide an explicit Euler ODE solver from Gaussian noise toward physically plausible human motion trajectories.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Arbitrary Wearable Subsets<br/>Phones / Watches / Headset / Insoles / Action"] --> B["Multimodal Calibration & Feature Projection<br/>Local frame processing + per-modality MLPs"]
    B --> C["Conditional Flow-Matching Backbone<br/>8-block DiT + adaLN time/action modulation"]
    C --> D["Modality-Dedicated Dynamic Cross-Attention<br/>Scaled aggregation over active sensor subset"]
    D --> E["ODE Integration Inference<br/>Euler trajectory solver generating 3D motion"]

Key Designs

1. Multimodal Calibration & Feature Projection: Aligning Heterogeneous Wearables in Local Frames

Consumer wearables lack centralized hardware clocks, and their onboard magnetometers exhibit severe magnetic drift in indoor settings. Rather than relying on fragile global heading estimates, WHIP operates strictly in device-local coordinate frames: smartphones (front left/right trouser pockets) and smartwatches (left/right wrists) provide local orientation \(R_{D,t}^S \in \mathrm{SO}(3)\), angular velocity \(\omega\), and linear acceleration \(a\); the Meta Quest 3 headset streams 6-DoF head pose \(H_t \in \mathrm{SE}(3)\); and Moticon OpenGo insoles record 16-zone plantar pressure distributions \(p_t^S \in \mathbb{R}^{16}\) alongside raw embedded IMU signals. The system solves spatial extrinsic transformations \(R_c, R_r\) and temporal clock offsets \(t_0\) through an initial excitation routine, and compensates for pocket shifting via 20-second block-level re-calibration. Each sensor stream is independently mapped into a 64-dimensional sequence of sensor tokens \(\mathbf{c}_t^m = \mathrm{MLP}_m(\mathbf{s}_t^m)\).

2. Conditional Flow-Matching Backbone: Continuous Probability Paths for Ambiguous Sparse Sensing

Inferring 3D full-body motion from sparse wearable cues is fundamentally underconstrained—a static head pose can correspond to multiple feasible leg configurations. Deterministic regression baselines typically collapse into over-smoothed poses or exhibit foot-sliding artifacts. WHIP formulates motion generation under the continuous-time Flow Matching framework, learning a velocity field \(u_\tau(X_\tau, C)\) that transports a prior Gaussian distribution \(X_0 \sim \mathcal{N}(0, I)\) into the target data distribution \(X_1 \sim p_1(\cdot|C)\). Adopting a conditional linear probability path:

\[u_\tau(X_\tau | X_1, C) = \frac{X_1 - X_\tau}{1 - \tau}\]

The network employs an 8-layer Diffusion Transformer (DiT) with latent dimension \(d=768\) and 12 attention heads. Frame tokens explicitly split into motion and sensor-reserved subspaces \(x_t = [x_t^{\text{mot}}; x_t^{\text{sens}}]\). Sequence-level flow time \(\tau \in [0, 1]\) and action embeddings modulate the transformer layers via adaLN, steering the velocity field throughout diffusion trajectories.

3. Modality-Dedicated Dynamic Cross-Attention: Plug-and-Play Conditioning for Arbitrary Subsets

To support unpredictable missing sensors without requiring zero-padding or model retraining, WHIP assigns an independent cross-attention module \(\text{CrossAttn}_m\) to each physical sensor instance (headset, insoles, left/right watches, and left/right phones). At inference time, absent sensors are completely bypassed without incurring computation or corrupting latent states. For all active modalities \(m \in \mathcal{M}\), motion tokens serve as queries against sensor token sequences \(\mathbf{C}^m\). To preserve residual activation variance regardless of how many devices are worn, WHIP incorporates an adaptive square-root scaling normalization factor:

\[\mathbf{X}' = \mathbf{X} + \frac{1}{\sqrt{|\mathcal{M}|}} \sum_{m \in \mathcal{M}} \text{CrossAttn}_m(\mathbf{X}, \mathbf{C}^m)\]

During training, an independent sensor dropout probability of 0.5 exposes the network to the entire combinatorial distribution of sparse inputs, ensuring exceptional robustness and graceful degradation from fully instrumented setups down to single-sensor cases.

Loss & Training

The network processes motion sequences using sliding temporal windows of \(T = 90\) frames (3 seconds at 30 fps) with a batch size of 64. The model optimizes the conditional flow matching mean squared error loss:

\[\mathcal{L}(\theta) = \mathbb{E}_{\tau, X_0, X_1, C} \left\| u_\theta(X_\tau, \tau, C) - \frac{X_1 - X_\tau}{1 - \tau} \right\|^2\]

Optimization uses AdamW with an initial learning rate of \(10^{-3}\) and a Warmup-Stable-Decay (WSD) schedule over 110k steps. At inference time, starting from \(X_0 \sim \mathcal{N}(0, I)\), trajectories are integrated across \(\tau \in [0, 1]\) via an explicit Euler solver with \(N\) discretization steps. To ensure a rigorous, unbiased comparison against deterministic regression baselines without speculative best-of-K selection, WHIP adopts the Bayes estimator under L2 loss by evaluating the empirical posterior mean across \(K = 10\) generated samples: \(\hat{X} = \frac{1}{K} \sum_{k=1}^K X_k\).

Key Experimental Results

Main Results

WHIP is evaluated on held-out test splits across two generalization dimensions: unseen participants (Unseen Actor) and unseen action categories (Unseen Action). Evaluated metrics comprise Mean Per Joint Position Error (MPJPE, mm, root-aligned), Mean Root Error (MRE, mm), scale-normalized MPJPE (N-MPJPE, mm), and Procrustes-aligned MPJPE (PA-MPJPE, mm). Baselines include parameter-matched MLP and Transformer regressors operating on zero-masked concatenated latents.

Evaluation Split	Method	Params	MPJPE ↓	MRE ↓	PA-MPJPE ↓	N-MPJPE (All) ↓	N-MPJPE (HMD) ↓	N-MPJPE (Ins) ↓	N-MPJPE (W+P Avg) ↓	N-MPJPE (Avg All) ↓
Unseen Actor	MLP Baseline	54.6M	127.8	147.5	95.0	203.2	157.0	123.8	120.1	120.1
	Transformer Baseline	57.4M	78.3	135.9	61.8	125.9	147.6	98.9	93.9	93.9
	WHIP (Ours)	59.4M	56.7	107.4	46.5	54.6	151.7	83.0	77.9	77.9
Unseen Action	MLP Baseline	54.6M	128.5	140.9	106.6	146.5	142.3	128.7	125.1	125.1
	Transformer Baseline	57.4M	83.3	140.5	74.1	80.3	130.6	115.8	106.0	106.0
	WHIP (Ours)	59.4M	60.0	106.6	55.7	37.5	121.8	99.5	87.0	87.0

When retrained against prior specialized state-of-the-art methods (Tab. 3, N-MPJPE, mm), WHIP achieves 61.7 mm in the IMU-only setting (phones and watches), significantly outperforming IMUPoser (75.5 mm). In HMD-centric setups, WHIP reaches 52.6 mm under full instrumentation, outperforming HMDPoser (74.6 mm) and AvatarPoser (78.2 mm with headset and dual watches).

Ablation Study

To rigorously quantify cross-modal interactions, the authors exhaustively evaluate all \(2^7 = 128\) sensor subsets, calculating the marginal error reduction score \(\text{Marginal}(m) = \mathbb{E}_{S} [v(S) - v(S \cup \{m\})]\) and pairwise synergy \(I(a, b) = \mathbb{E}_{X} [v(X_{ab}) - v(X_a) - v(X_b) + v(X)]\) on N-MPJPE.

Sensor Configuration / Modality	Marginal Score / Interaction Dynamics	Note & Empirical Analysis
HMD (Head-Mounted Device)	Largest marginal error reduction (~30-40 mm)	Uniquely restores absolute metric world scale; contribution narrows under N-MPJPE
Smartwatches	Second-largest overall marginal contributor	Dominates upper-limb accuracy, resolving elbow and wrist kinematic ambiguities
Phones + Insoles	Highest pairwise synergy (\(I(a, b) \approx -5 \text{ mm}\))	Phone captures pelvis/thigh angles while insoles ground contact events; strong lower-body synergy
2× Phone or 2× Watch	High redundancy (\(I(a, b) \approx -20 \text{ mm}\))	Symmetrical on-body IMUs exhibit substantial informational overlap during standard gait
Pareto Frontier (k=1, 2, 3)	k=1: HMD; k=2: Watch+Phone; k=3: HMD+Watch+Phone	Watch+Phone outperforms HMD-inclusive pairings at k=2 by bridging upper and lower extremities

Key Findings

• Generative Flow Matching Outperforms Regression: On the unseen actor benchmark with all sensors active, WHIP attains an MPJPE of 56.7 mm compared to 78.3 mm for the transformer regressor (a 27.6% error reduction), proving that continuous flow modeling effectively resolves multimodal kinematic ambiguity and eliminates over-smoothing.
• The Non-Obvious Pareto Optimum at Two Devices: While the HMD is the single strongest device (\(k=1\)), the two-device Pareto optimum is Watch+Phone rather than any HMD pairing. Spanning upper and lower extremities offers far richer kinematic coverage than concentrating sensing in the upper body.
• Plantar Insoles as Ideal Grounding Compensators: Although smart insoles yield modest stand-alone tracking, they exhibit the highest pairwise complementarity (\(I(a, b)\) closest to zero) when combined with other sensors, drastically eliminating footskate artifacts.

Highlights & Insights

• Adaptive Scaled Modality-Specific Cross-Attention: The \(1/\sqrt{|\mathcal{M}|}\) scaling stabilizes residual update variances dynamically across all 128 sensor subsets, allowing a single unified model to gracefully handle extreme sparsity down to a single watch or phone.
• First Systematic Cooperative Sensor Study in Consumer MoCap: Leveraging cooperative game theory and marginal error reductions, the study establishes an empirical foundation for how commodity sensors interact, revealing where hardware redundancy occurs and how different body parts complement one another.
• Strong Generalization Beyond the Capture Studio: Despite training on 14 laboratory subjects, WHIP transfers zero-shot to the Nymeria dataset and unconstrained in-the-wild recordings, validating the robustness of the learned motion prior.

Limitations & Future Work

• Omission of Fine-Grained Hand Tracking: The dataset currently omits VR hand controllers or finger-tracking signals, leaving dexterous hand-object interactions unmodeled.
• Numerical Integration Latency for Real-Time Use: Because inference relies on multi-step Euler ODE integration, WHIP exhibits non-trivial compute latency, preventing immediate real-time execution on resource-constrained smartwatches or smartphones.
• Reliance on Windowed Calibration for Pocket Shift: Accommodating phone displacement inside trouser pockets currently requires 20-second recalibration blocks; future work should incorporate dynamic on-body calibration modules for fully autonomous tracking.

Related Work & Insights

• vs IMUPoser / MobilePoser: While IMUPoser relies on discrete classification or shallow regressors that degrade under missing modalities, WHIP offers a continuous generative flow-matching formulation that processes arbitrary sensor subsets with superior temporal smoothness and physical plausibility.
• vs AvatarPoser / HMDPoser: AvatarPoser and HMDPoser are rigidly tethered to HMD-centric rigs and fail when the headset is removed; WHIP treats the HMD as an optional input modality, gracefully reconstructing motion from pure phone/watch setups.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Establishes the first synchronized consumer-grade MoCap benchmark across phones, watches, insoles, and HMD, coupled with an adaptive flow-matching architecture
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exhaustively evaluates all 128 sensor combinations with marginal and interaction analyses, complemented by cross-dataset and in-the-wild evaluations
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear architectural rationale, and insightful empirical analysis
• Value: ⭐⭐⭐⭐⭐ Sets a new standard for commodity wearable motion capture and offers invaluable design guidance for next-generation pervasive tracking systems