ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control

Conference: ECCV2026
Paper: Official page · PDF
Project: ConTrack
Area: Video Understanding / Robotic Dexterous Manipulation
Keywords: human-to-robot retargeting, constrained reinforcement learning, task-style trade-off, mid-trajectory resets, contact priors

TL;DR

ConTrack treats object motion tracking as the priority task constraint and learns robotic hand control with online weight adaptation, a reachable-state reset library, and contact priors, achieving mean progress of 0.899 and contact F1 of 0.784 within 5000 PPO updates per clip, without leading every error metric.

Background & Motivation

Learning dexterous manipulation from human demonstrations involves more than mapping human joint angles onto robot joints. Differences in finger lengths, joint freedoms, and actuation mean that the same geometric pose may not establish the same contacts on a robot; even a visually similar hand configuration can let an object slip. Object trajectories describe whether the task is accomplished, whereas finger configurations and contact timing describe fidelity to the demonstration. These objectives are not always jointly achievable.

Existing approaches use residual control, object-centric objectives, or physically feasible retargeting, but still need to decide how much imitation accuracy to sacrifice when a reference cannot be executed. Fixed reward weights struggle across clips and contact phases. Training distribution adds another difficulty: repeatedly starting from the first frame leaves later phases undertrained, while jumping directly to a reference middle frame can initialize mutually inconsistent contacts and an unrecoverable simulator state.

ConTrack therefore controls both the allocation of optimization effort and the starting points of training. Core idea: define a relative tracking requirement using historically attainable task performance, adapt task-style optimization weights online, and restart difficult segments from physical states actually visited by the policy so that object motion succeeds while joint and contact fidelity are preserved as far as possible.

Method

Overall Architecture

The inputs are retargeted robot joint references, object pose trajectories, link-level contact events, and contact points expressed in object-local coordinates. Each reference clip defines a finite-horizon task whose state includes the simulator physical state and reference frame index. The paper trains one policy per clip, not a general video-to-action model shared across clips.

The policy predicts residual displacements around reference joint positions; adding the residual to the reference produces the execution target. Task rewards measure object poses, style rewards measure hand motion and contacts, and a separate term penalizes high-frequency motion. Adaptive task-style mixing selects the advantage signals used by PPO, the reachable-state reset library selects rollout starts, and contact priors refine style supervision. Simulator rollouts provide training feedback to these mechanisms.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Joint, object, and contact references"] --> B["Adaptive task-style mixing"]
    A --> C["Reachable-state reset library"]
    A --> D["Contact priors"]
    B --> E["PPO residual policy<br/>and simulator rollouts"]
    C --> E
    D --> E
    E -->|Task return| B
    E -->|Reachable states and continuation lengths| C
    E --> F["Physically executable joint trajectories"]

Key Designs

1. Adaptive task-style mixing: secure the object task before optimizing motion fidelity

The method maximizes style return within a task constraint. With discounted task and style returns denoted by \(J_g\) and \(J_s\), the objective is to maximize \(J_s\) subject to \(J_g\geq\alpha J_g^\star\). Here \(\alpha\in(0,1]\) is a chosen target ratio, and \(J_g^\star\) is the historical maximum of a running task-return estimate, not a known optimal return. Normalization makes constraints comparable across clip scales, but also means that the mechanism is practical trade-off control rather than a strict task-success guarantee.

Training collects rollouts, updates the running task-return estimate \(\hat J_g\) and its historical maximum, and then updates a scalar controller state \(\lambda\). When the ratio of current to historically best return falls below the target, task weight increases; when it exceeds the target, task weight decreases and releases optimization capacity for style. The weight is shared by all parallel environments for a clip rather than independently predicted for each contact.

\[ \lambda\leftarrow\lambda+\eta\left(\alpha-\frac{\hat J_g}{J_g^\star}\right),\qquad w_{\mathrm{task}}=\sigma(\lambda). \]

The controller operates through advantage mixing, not merely a sum of raw rewards. Task, style, and penalty rewards provide separate advantage estimates. The penalty advantage remains independently active rather than disappearing with the task-style allocation:

\[ A_{\mathrm{mix}}=w_{\mathrm{task}}A_g+(1-w_{\mathrm{task}})A_s+A_p. \]

The cached extraction of body equations (5) and (7) loses symbols; the expressions above follow the explicit steps in Algorithm 1 and the adjacent explanation. The paper uses a real-valued \(\lambda\) mapped through a sigmoid, so it should not be silently replaced with a standard nonnegative projected Lagrange multiplier. The user still chooses \(\alpha\): reduced fixed-weight tuning does not mean an absence of hyperparameters.

2. Reachable-state reset library: rehearse difficult suffixes without inventing contact states

The library is indexed by reference frame \(k\) and stores robot joint positions and velocities together with object poses and velocities. After an episode ends, each visited frame receives a continuation length measuring how long execution lasted from that frame. If this length beats the frame's historical best, the library replaces its entry with the actual simulator state from that rollout. It stores a state visited by the policy that supports longer continuation, not a snapshot copied from reference geometry.

State selection and frame sampling use different statistics: the former compares historical maximum continuation lengths, whereas the latter uses their exponential moving averages. Dividing average continuation by the remaining horizon gives a survival ratio, and frames with lower survival ratios receive greater sampling probability:

\[ \ell_k=t_{\mathrm{end}}-k+1,\qquad u_k=\frac{\bar\ell_k}{T-k},\qquad p(k)\propto\exp\left(-\frac{u_k}{\tau}\right). \]

Here \(\bar\ell_k\) is an exponential moving average updated with coefficient \(\beta\), and \(\tau>0\) is a sampling temperature. Normalization matters because later frames naturally have fewer remaining steps; absolute continuation alone would misrepresent their difficulty. Training can first master shorter suffixes and then shift toward earlier failure boundaries. This is not a hard-coded frame-by-frame reverse curriculum but an outcome of state-quality updates and difficulty-aware sampling.

Algorithm 2 specifies online updates, but the available cache does not clearly explain library initialization for unvisited frames or provide concrete \(\beta\) and \(\tau\) settings. No initialization scheme is invented here; reproduction still requires complete supplementary material or code.

3. Contact priors: constrain which link contacts where, not only object motion

Object pose supervision alone can allow a policy to use very different finger combinations from the demonstration. ConTrack maps dataset contact annotations to robot links and rewards agreement between simulated and reference contact events. Only when both indicate contact does it additionally penalize the distance between their contact points. Points are expressed in object-local coordinates so that global object motion does not obscure the comparison of contact locations.

These signals answer complementary questions: whether contact occurs at the appropriate time, and whether it occurs at the intended location. They belong to the style objective and are indirectly regulated by adaptive mixing, rather than imposing additional hard contact constraints. The paper describes this supervision logic, but the cache omits complete reward kernels, coefficients, and the annotation-processing appendix. A chosen distance-reward implementation should not be presented as the authors' exact implementation.

A Worked Example

Consider GRAB waterbottle_offhand, a 4.0-second bimanual rigid-object interaction. The policy produces residuals around reference joint positions. If object-tracking return deteriorates, the controller increases task weight, permitting fingers to depart from the reference to maintain object motion. Contact priors continue to constrain link contact events and locations so that moving the object does not entirely replace the demonstrated interaction pattern.

If an intermediate contact phase repeatedly fails, the library reallocates training starts using visited states and continuation statistics. A longer continuation from a frame then refreshes that frame's cached state. Final evaluation still begins at the first frame: Table 3 reports progress of 1.000, object translation error of 0.018 m, and contact F1 of 0.745 for this clip. This walkthrough illustrates the mechanism; the paper does not report stepwise \(\lambda\) values for this clip.

Loss & Training

PPO updates the policy using the mixed advantage, with the high-frequency-motion penalty kept separate. Each clip receives 5000 PPO updates. Learning-based comparisons use a fixed simulator-step budget; mid-trajectory resets redistribute samples without increasing total interactions. Network architecture, learning rate, controller step size \(\eta\), and the complete reward implementation are not specified in the available cache and cannot be filled using generic PPO defaults.

Hardware validation does not directly deploy a full perception-driven closed-loop policy. Simulator-predicted joint references are streamed over TCP to a separate real-time controller on two xArm7 arms and two xHands. Each side tracks 7 arm joints and 12 hand joints, with outer reference updates every 0.15 s and an internal arm loop at 250 Hz.

Key Experimental Results

Main Results

Evaluation covers 5 GRAB clips, 4 ARCTIC clips, and 4 DexterHand clips, spanning bimanual rigid objects, articulated objects, and continuous single-hand rotation. Episodes terminate early if any object exceeds 0.10 m translation error or 1.00 rad rotation error. All evaluations start from the first reference frame.

Progress is the termination-frame index divided by \(T-1\), not success rate; SR is the fraction of episodes reaching the final frame. Object and joint errors are averaged only over executed frames. Contact F1 comes from an event confusion matrix aggregated over time and link-object pairs, while contact-point error is evaluated only on matched contacts. Low error from an early-terminating method must therefore be interpreted alongside progress.

The following reproduces the selected comparison from Table 2. Values are means ± standard deviations across clips, not confidence intervals across random seeds.

Method	Progress ↑	Object position error m ↓	Object rotation error rad ↓	Finger error rad ↓	Contact F1 ↑	Contact point error m ↓
ConTrack	0.899 ± 0.195	0.026 ± 0.006	0.272 ± 0.105	0.163 ± 0.014	0.784 ± 0.072	0.018 ± 0.005
ManipTrans	0.743 ± 0.292	0.012 ± 0.009	0.207 ± 0.078	0.277 ± 0.089	0.620 ± 0.068	0.030 ± 0.010
DexMachina	0.246 ± 0.052	0.038 ± 0.018	0.348 ± 0.121	0.147 ± 0.016	0.708 ± 0.041	0.024 ± 0.003
SPIDER	0.444 ± 0.341	0.201 ± 0.113	1.104 ± 0.599	0.157 ± 0.019	0.191 ± 0.225	0.036 ± 0.011

ManipTrans is reimplemented for the authors' environment. Because DexMachina is not fully open source, the comparison integrates its Virtual Object Controllers. SPIDER does not learn a policy. This is consequently not a direct ranking of original systems under identical training procedures. Against ManipTrans, ConTrack improves progress by 0.156 and contact F1 by 0.164, but has higher object position and rotation errors.

Ablation Study

The following collects mean values from Tables 4, 5, and 6 using the same full model as a reference. Row standard deviations are omitted here to simplify cross-mechanism reading.

Configuration	Progress ↑	Object rotation error rad ↓	Finger error rad ↓	Contact F1 ↑	Contact point error m ↓
Full model	0.899	0.272	0.163	0.784	0.018
Fixed task	0.764	0.250	0.157	0.679	0.022
Fixed task-style 1:1	0.868	0.297	0.165	0.701	0.023
Reset from start	0.700	0.370	0.168	0.739	0.017
Uniform reset	0.727	0.298	0.152	0.714	0.021
Without contact event reward	0.861	0.320	0.168	0.699	0.020
Without contact distance reward	0.868	0.288	0.149	0.753	0.023

Key Findings

• The reset library has the largest effect on progress: resetting only from the start reduces it from 0.899 to 0.700, an absolute decrease of 0.199. Coverage of later contact phases is crucial under limited interactions.
• Against fixed 1:1 mixing, adaptation raises progress from 0.868 to 0.899 and contact F1 from 0.701 to 0.784. Figure 4 sweeps \(\alpha=0.6,0.7,0.8,0.9\) and shows an empirical Pareto frontier within the tested range, not proof of global optimality.
• Removing the contact event reward lowers F1 to 0.699. Removing the contact distance reward raises contact-point error to 0.023 m while improving finger error to 0.149 rad. Closer reference joints do not necessarily produce better physical contacts.
• Mean progress across the three tiers is 0.996, 0.944, and 0.733; DexterHand Ring reaches only 0.272. The authors localize failure to a rotation-dominant phase and state that longer training resolves it, but the extended-budget appendix numbers are absent from the cache.

Highlights & Insights

• A relative task requirement gives task and style unequal priority. Improving imitation once the task reaches a currently attainable level is a more interpretable control interface than fixed weights.
• The reset library combines physical consistency with learning difficulty. Difficult-frame sampling alone can create unrecoverable starts, while good-state caching alone can repeat easy segments; their combination supports an effective curriculum.
• Contact events and object-local contact points provide complementary supervision. This can inform other cross-embodiment manipulation settings, provided reliable contact annotations are available.

Limitations & Future Work

• The authors acknowledge that running-maximum normalization does not strictly guarantee constraint satisfaction. A low historical best can make relative feasibility compatible with absolute task failure; absolute object-error or completion constraints are a possible extension.
• Policies are trained separately per clip, and evidence comes from a limited clip set rather than a general cross-object, cross-demonstration policy. Fixed-budget improvements should not be extrapolated into established large-scale generalization.
• Contact priors depend on annotation quality and link mapping, and the hardest Ring clip still fails substantially within the current budget. Noisy contact supervision, complex finger transitions, and domain differences deserve separate evaluation.
• Hardware results establish feasibility of executing joint references. The cache provides no real-robot success-rate table and does not establish closed-loop robustness to visual or tactile disturbances. Richer perception and tighter simulation-to-reality alignment are directions identified by the authors.
• The cache ends after the references and lacks the repeatedly cited appendix. Architecture, reset-library initialization, full hyperparameters, and extended-training results cannot be recovered from the available material.

Related Work & Insights

• Versus ManipTrans: Both allow residual corrections to demonstrations, but ConTrack emphasizes online task-style allocation and reset distributions. ManipTrans has lower object errors in the table; ConTrack executes more of the trajectory and better preserves contact patterns.
• Versus DexMachina: Object-centric functional retargeting emphasizes task outcomes, while ConTrack adds an explicit style objective and adaptive requirement. Interpretation is limited by the integrated baseline version and fixed budget.
• Versus SPIDER: Physics-informed retargeting produces feasible reference trajectories, whereas ConTrack learns feedback control in a simulator. Low joint error alone does not ensure stable long-horizon contact tracking.
• Connections to DeepMimic and Backplay: Reference-based restarts and suffix learning have precedents. The targeted change is caching policy-visited coupled hand-object states and allocating training by relative continuation ability, not inventing mid-trajectory resets themselves.

Rating

• Novelty: 4/5. Constrained trade-offs combined with reachable-state curricula fit contact-rich transfer well, although the underlying optimization ideas have precedents.
• Experimental Thoroughness: 3/5. Three task groups and ablations for all three mechanisms are useful, but per-clip training, adapted baselines, and missing hardware statistics limit the conclusions.
• Writing Quality: 4/5. The task-style conflict, controller, and curriculum are clearly explained, while reproduction still depends on an appendix absent from the cache.
• Value: 4/5. Practical ideas for turning human demonstrations into executable dexterous trajectories, but not a replacement for a general closed-loop robot policy.