HumanTracker: Towards Comprehensive and Human-Aligned Motion Tracking Benchmark¶

Conference: ECCV 2026
Paper: ECCV Official Page
Code: https://dairuliu.github.io/humantracker
Area: Alignment & RLHF
Keywords: humanoid motion tracking, evaluation benchmark, human preference alignment, reward model, physical contact realism

TL;DR¶

HumanTracker introduces a 150-hour optical mocap benchmark across four motion families and HumanScore, a temporal Transformer trajectory reward model trained on 12K human preference pairs, achieving 92.91% preference alignment and revealing contact failures missed by kinematic metrics.

Background & Motivation¶

Humanoid whole-body motion tracking in physics simulation is the cornerstone for teleoperation and whole-body imitation learning. Trackers typically optimize a learned feedback policy to follow kinematic reference trajectories under closed-loop control. However, measuring tracking fidelity reveals a fundamental discordance between numerical evaluations and human perception: rollouts with low kinematic errors such as Mean Per-Joint Position Error (MPJPE) can still look unnatural, exhibiting glaring physical artifacts like foot skating, mistimed touch-downs, high-frequency joint jitter, and sudden loss of balance. Conventional kinematic metrics evaluate per-frame geometric pose matching and average errors uniformly across joints and frames, ignoring support polygon dynamics, contact transitions, and the cumulative divergence characteristic of closed-loop physics.

Concurrently, evaluation is bottlenecked by the limited scope and diversity of existing benchmarks. Despite large-scale repositories such as PHUMA and SONIC, prevailing humanoid tracking benchmarks rely primarily on an AMASS evaluation split of only 140 sequences. This compact test set is dominated by basic locomotion, leaving the long tail of human movement—such as aerial impact landing, asymmetric balancing, and contact-rich ground recovery—heavily underrepresented. Furthermore, summarizing tracker capability via a single global average obscures why and under which dynamic regimes a controller breaks down.

The authors address this gap by establishing an evaluation paradigm grounded in human perceptual judgment and providing an extensive, diagnostically categorized benchmark. Core idea: construct HumanTracker, a 150-hour high-fidelity optical mocap benchmark structured into four distinct motion families, and train HumanScore, a temporal Transformer reward model over 12K paired rollout comparisons, to directly predict human perceptual preference and penalize contact-timing and stability failures.

Method¶

Overall Architecture¶

The HumanTracker evaluation system comprises two complementary pillars: a 150-hour optical motion capture dataset covering 24K clips categorized for fine-grained failure diagnosis, and a trajectory-level reward model (HumanScore) trained on human preferences. To evaluate a humanoid tracking policy, rollouts are executed in a standardized physics simulation against retargeted reference trajectories (qpos format). Tracking trajectories are segmented into non-overlapping 5-second windows. For each window, per-frame rollout state vectors (kinematic states and privileged contact dynamics) concatenated with time-aligned reference features are processed by a 5-layer bidirectional Transformer encoder. A global mean pooling layer aggregates frame tokens into a sequence embedding, which is projected by an MLP head to a scalar preference reward $r_\theta(\tau)$. Finally, raw segment scores are averaged and normalized via a global sigmoid transformation onto a standardized 0–100 scale.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference Motion & Policy Rollouts<br/>Unified qpos and standardized physics simulation"] --> B["Multi-Dimensional Temporal Feature Extraction<br/>Kinematic states, contact forces, and reference alignment"]
    B --> C["Bidirectional Temporal Transformer Encoder<br/>5-layer multi-head self-attention capturing dynamics"]
    C --> D["Global Mean Pooling & Reward Prediction Head<br/>Aggregating temporal tokens to scalar reward r(τ)"]
    D --> E["Bradley-Terry Preference Optimization<br/>Pairwise cross-entropy with margin and tie regularization"]
    E --> F["HumanScore Calibration & Diagnostic Evaluation<br/>0-100 calibrated metric across four motion families"]

Key Designs¶

1. Multi-Dimensional Temporal Feature Extraction: Fusing Privileged Dynamics with Reference Cues Traditional evaluation relies solely on external visual geometry, failing to directly register ground reaction forces and support polygon transitions. To eliminate visual rendering confounds, HumanScore processes the underlying physical time series rather than raw video pixels. At each timestep $t$, a feature vector $x_t$ concatenates the humanoid's full-body joint positions, linear and angular velocities, time-aligned reference state features $q_t^{\mathrm{ref}}$, and privileged simulator signals—including foot contact flags, ground contact forces, height above ground, and foot slip velocities. These features are projected through a linear layer and LayerNorm into $D_{\mathrm{model}}=1024$, combined with sinusoidal positional encodings to establish a comprehensive representation of physical contact and tracking discrepancy.

2. Bidirectional Temporal Transformer Encoder: Capturing Long-Horizon Stability and Contact Timing Physical artifacts such as foot sliding, ground jitter, and unstable support switching unfold over multi-second temporal windows, rendering single-frame assessment insufficient. HumanScore leverages a 5-layer bidirectional Transformer encoder with 2 attention heads and a feed-forward dimension of 1024. The bidirectional self-attention mechanism correlates events across the full 5-second window, enabling the model to explicitly reason about swing-to-stance transitions, landing impact phases, and center-of-mass balance over extended horizons.

3. Global Mean Pooling and Reward Head: Sequence-Level Perceptual Scoring Across the encoded sequence of $T$ frame tokens, the model applies global mean pooling to construct a single vector representation, which is mapped through a multi-layer perceptron (MLP) head to produce a scalar reward $r_\theta(\tau) \in \mathbb{R}$. For a full rollout $\tau$ partitioned into $N$ non-overlapping 5-second segments $\{s_i\}_{i=1}^N$, the unnormalized trajectory score averages the segment rewards: $$\mathrm{Score}_{\mathrm{RM}}^{\mathrm{raw}}(\tau) = \frac{1}{N} \sum_{i=1}^N r_\theta(s_i)$$ To provide an interpretable 0–100 scale across experimental settings $b$ (in-domain or zero-shot), scores are normalized via setting-level median and quartile dispersion metrics $\mathrm{scale}_b = \max((q_{75,b}-q_{25,b})/(2\log 3), \epsilon)$: $$\mathrm{HumanScore}_b = 100 \cdot \sigma\left(\frac{\mathrm{Score}_{\mathrm{RM}}^{\mathrm{raw}} - \mathrm{median}_b}{\mathrm{scale}_b}\right)$$

4. Bradley-Terry Preference Optimization: Reconciling Strict Choices and Ties The model is trained on 12K human-annotated video pairs rendered from synchronized rollouts of trackers trained with varying reward shaping and domain randomization. Annotators selected from four outcomes: Left better, Right better, Both good, or Both bad. For strict preference pairs $(\tau_{\mathrm{chosen}}, \tau_{\mathrm{rejected}})$, the objective minimizes the negative log-likelihood under the Bradley-Terry model with margin $m$: $$\mathcal{L} = -\log \sigma(r_\theta(\tau_{\mathrm{chosen}}) - r_\theta(\tau_{\mathrm{rejected}}) - m)$$ For tied pairs (Both good and Both bad), a score-consistency loss penalizes absolute score divergence between the two rollouts, while an absolute anchoring loss calibrates the score scale by constraining Both good pairs to attain higher baseline rewards than Both bad pairs.

Loss & Training¶

The trajectory reward model is fine-tuned for 4 epochs using AdamW with batch size 128, initial learning rate $1\times 10^{-5}$ regulated by a cosine annealing schedule, dropout 0.05, and weight decay $1\times 10^{-5}$. The 12K preference pairs are partitioned into an 8:2 train/test split. Evaluation is performed in a standardized rigid-body simulation environment under uniform control frequency, episode termination criteria, and action bounds to ensure unbiased benchmarking.

Key Experimental Results¶

Main Results¶

On the held-out preference test split, HumanScore demonstrates superior alignment with human judgments compared to traditional kinematic and smoothness metrics:

Metric / Model	Align Rate ↑	Note
HumanScore (Reward Model)	0.9291	Proposed temporal Transformer reward model
MPJPE (rad)	0.8218	Mean per-joint pose error
Avg Joint Jerk (rad/s³)	0.7088	Third-order joint smoothness metric
Avg Joint Accel (rad/s²)	0.6686	Second-order joint acceleration
KPT Position MAE (m)	0.5594	Keypoint position mean absolute error
Foot Contact Accuracy	0.5307	Heuristic binary foot contact accuracy
MPJVE (rad/s)	0.4770	Mean per-joint velocity error

In the zero-shot evaluation across four motion families on HumanTracker, state-of-the-art trackers exhibit substantial divergence across distinct contact regimes:

Method	Daily (Succ % / MPJPE / HumanScore)	High Dynamic (Succ % / MPJPE / HumanScore)	Interaction (Succ % / MPJPE / HumanScore)	Ground (Succ % / MPJPE / HumanScore)
Humanoid-GPT	97.2% / 0.060 / 72.0	89.9% / 0.102 / 64.0	97.6% / 0.051 / 70.9	19.2% / 0.608 / 23.3
SONIC	93.6% / 0.103 / 52.5	92.3% / 0.118 / 43.1	96.6% / 0.080 / 65.9	0.0% / 0.260 / 35.3
TWIST2	80.0% / 0.106 / 24.3	62.3% / 0.164 / 21.8	93.5% / 0.075 / 41.2	0.0% / 0.320 / 13.3
GMT	21.1% / 0.244 / 12.7	30.7% / 0.250 / 16.2	76.7% / 0.151 / 40.2	3.0% / 0.436 / 15.7

Ablation Study¶

Ablations on the held-out preference dataset evaluate inference-time feature zeroing and temporal context lengths:

Configuration / Feature Group	Align Rate	Note
Full Model (5s context)	0.9291	All kinematic, reference, and contact features
w/o Contact (zero contact-related features)	0.7395	Drastic 18.96% drop; contact cues dominate human judgment
Kinematics only (only joint pose features)	0.7548	Pose errors alone fail to capture perceived physical plausibility
w/o Reference (zero reference trajectory features)	0.9253	Minor reduction; physical plausibility outweighs precise tracking
w/o Root Vel (zero root velocity features)	0.9291	Model is insensitive to global root velocity
Temporal Context Length
1s Context	0.7739	Insufficient context to capture gait cycles or transitions
2s Context	0.8525	Basic periodic motion captured; notable gain
3s Context	0.8812	Approaching steady-state alignment
4s Context	0.9138	Improved identification of cumulative instability
5s Context (Default)	0.9291	Complete temporal context for contact and balance assessment

Key Findings¶

Physical Contact is the Principal Driver of Perceptual Preference: Zeroing contact features precipitates an Align Rate collapse from 0.9291 to 0.7395, and retaining only kinematic features yields 0.7548. Human observers are acutely sensitive to foot skating, ground penetration, and unstable support switches, which are diluted in spatial pose averages.
Extended Temporal Context is Critical: Align Rate scales monotonically from 77.39% at 1s to 92.91% at 5s. Stumble recovery, drift, and dynamic stability are inherently time-extended processes that cannot be diagnosed from instantaneous frames.
Ground-Level Motions Remain an Open Frontier: Across zero-shot evaluations, trackers like SONIC and TWIST2 completely fail on the Ground family (0.0% success rate), and Humanoid-GPT reaches only 19.2%. Managing low-posture multi-contact transitions (sitting, kneeling, rolling) represents the primary unresolved challenge for humanoid control.

Highlights & Insights¶

From Geometric Fitting to Perceptual Alignment: Transfers RLHF-inspired preference reward modeling into humanoid motor control evaluation, establishing HumanScore as a reliable scalar metric that reconciles kinematic optimization with visual and physical realism.
Taxonomy-Driven Diagnostic Benchmarking: Structures 150 hours of optical mocap into Daily, High Dynamic, Interaction, and Ground families, replacing aggregate benchmarks with regime-specific failure analysis.
Potential for Policy Optimization: Demonstrates that while HumanScore is primarily an evaluation metric, it can serve as a dense reward during PPO training to mitigate foot skating and jitter.

Limitations & Future Work¶

Susceptibility to Reward Hacking in RL: Directly optimizing HumanScore via PPO can lead policies to adopt overly rigid postures to avoid contact penalties, necessitating careful regularization against task goals.
Sim-to-Real Perceptual Gap: Preferences are sourced from simulated video rollouts. Real-world physical effects such as gear backlash, torque saturation, and non-rigid terrain contact are not yet incorporated.
Rigid Ground Assumption: The benchmark currently focuses on flat rigid surfaces, leaving deformable terrain and complex multi-agent physical interactions for future extensions.

vs AMASS / PHUMA / SONIC: AMASS tracking splits (140 clips) lack scale and contact complexity. HumanTracker expands optical capture to 150 hours (24K clips) with manual physical plausibility cleaning, four-family taxonomy, and textual descriptions.
vs Kinematic Metrics (MPJPE / Keypoint MAE): Frame-wise pose metrics treat tracking as geometric matching and ignore support stability; HumanScore achieves 0.9291 preference alignment, resolving metric divergence.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Introduces a pairwise preference-aligned reward model (HumanScore) and a comprehensive 150-hour categorized optical mocap benchmark for humanoid tracking.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 12K annotated pairs, 4 state-of-the-art trackers, in-domain vs. zero-shot settings, and extensive feature/temporal ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with precise formulations, clear motivation, and thorough documentation.
Value: ⭐⭐⭐⭐⭐ Establishes a foundational, standardized benchmark and human-aligned evaluation metric for humanoid robotics, teleoperation, and physics-based imitation.