Skip to content

Exo2EgoPolicy: Pose-Aligned Cross-View Policy Learning

Conference: ECCV 2026
Paper: ECCV Official
Area: Robotics & Embodied AI
Keywords: Cross-View Policy Transfer, Pose Diffusion Model, SE(3) Manifold, Temporal Regularization, Embodied Co-Training

TL;DR

Exo2EgoPolicy bypasses pixel-level video hallucination by formulating exocentric-to-egocentric translation directly as an SE(3) hand-object pose diffusion process, combining temporal regularization and object-relative frames to enable scalable robot policy co-training with 20–30% higher success rates.

Background & Motivation

Learning generalizable robot manipulation policies from Internet-scale human demonstration videos offers a compelling route to mitigate the severe data scarcity in robotics. Through mirror-neuron mechanisms, humans naturally observe actions executed from an external viewpoint and project them into an internal, first-person motor representation. However, the overwhelming majority of accessible human videos are recorded from exocentric (third-person) viewpoints, whereas robot manipulation policies operate within an egocentric (first-person) frame of reference. Transferring manipulation knowledge across these disparate perspectives poses an ill-posed inverse problem characterized by extreme viewpoint shifts, uncalibrated moving cameras, and severe occlusions of critical hand-object contact interfaces.

Prior research generally tackles this challenge via novel view synthesis or generative video diffusion. NeRF-style geometric regression frameworks rely on dense multi-view coverage and fail under single-view inputs with heavy occlusion. On the other hand, pixel-space diffusion models hallucinate missing scene regions, frequently producing geometric drift, structural deformities, and severe frame-to-frame high-frequency flickering. When continuous robot control policies ingest these noisy, jittery pixel hallucinations, the kinematic inconsistencies disrupt velocity and acceleration profiles, deteriorating derivative-based low-level motor execution. Fundamentally, this stems from a misalignment in problem formulation: robotic manipulation policies depend primarily on the relative spatial and kinematic geometry between the hand and the object rather than photorealistic background textures.

To resolve this contradiction, the authors propose a paradigm shift from pixel synthesis to structured geometric alignment on the pose manifold. Formulating manipulation as a Markov Decision Process (MDP), the paper proves that for quasi-static manipulation tasks, relative hand-object pose serves as a sufficient statistic for optimal policy learning. Core idea: formulate exocentric-to-egocentric translation as a viewpoint-conditioned latent diffusion model directly over the SE(3) manifold, using object-centric reference frames to eliminate camera extrinsics ambiguity and applying a contractive Lie-algebra temporal regularizer to produce smooth, control-ready trajectories for unified policy co-training.

Method

Overall Architecture

The Exo2EgoPolicy framework consists of four integrated modules: geometric latent feature encoding from monocular exocentric video, viewpoint-conditioned pose diffusion over the SE(3) manifold, contractive temporal dynamics regularization, and unified human-robot policy co-training. The pipeline processes uncalibrated third-person video frames alongside initial coarse hand-object poses into dense geometric latent vectors. Operating in an object-relative local coordinate frame, a continuous-time score network reverses a Wiener noise process to recover first-person pose trajectories. A forward kinematic motion prior subsequently regularizes frame-to-frame transitions via Lie algebra geodesic distances, guaranteeing bounded cumulative error. Finally, the smooth egocentric trajectories are combined with limited robot teleoperation data in a shared state-action space to train an Action Transformer policy.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Exocentric Video + Coarse Poses"] --> B["Geometric Latent Encoder<br/>DINOv2 + MLP feature projection"]
    B --> C["SE(3) Object-Relative Pose Diffusion<br/>Marginalizing camera extrinsics via local frame"]
    C --> D["Temporal Dynamics Regularization<br/>Lie algebra difference & contractive error bound"]
    D --> E["Unified State-Action Co-Training<br/>Joint behavioral cloning with robot teleoperation"]
    E --> F["Dexterous Bimanual Robot Policy"]

Key Designs

1. SE(3) Object-Relative Pose Diffusion: Marginalizing Camera Extrinsics Ambiguity Monocular third-to-first view translation is fundamentally underdetermined due to unknown camera extrinsics and scale ambiguity. Predicting absolute poses in world coordinates forces the network to simultaneously disentangle camera trajectory and fine-grained contact mechanics, leading to substantial geometric drift. Exo2EgoPolicy overcomes this by anchoring the generation target directly to the local reference frame of the manipulated object: $\(p_t = (p_t^{\text{obj}})^{-1} \cdot p_t^{\text{hand}} \in \mathcal{M}_{\text{pose}}\)$ Here \(\mathcal{M}_{\text{pose}} \triangleq \mathcal{M}_{\text{hand}} \times \mathcal{M}_{\text{obj}}\), where hand pose comprises 6-DoF wrist transform plus \(K\)-dimensional articulation parameters, and object pose is in SE(3). Because the object serves as a locally static anchor, the dynamic camera motion is marginalized out. The generative reverse-time SDE operates over continuous pseudo-time \(\tau \in [0, 1]\) conditioned on dense geometric latent vectors \(z_t = \{E_i(i_t^{\text{exo}}); E_p(p_t^{\text{exo}})\}\) extracted via DINOv2 and an MLP, bypassing visual pixel hallucinations while remaining identifiable up to a rigid transformation.

2. Lie Algebra Forward Motion Prior: Enforcing Temporal Stability and Non-Divergence Independent frame-by-frame diffusion generates geometrically plausible static states but exhibits high-frequency kinematic jitter across consecutive timesteps. To establish strict temporal continuity, the score network conditions autoregressively on the prior state \(p_{t-1}\). In parallel, a learned forward transition prior \(\Phi(p_{t-1}, z_t)\) predicts the expected next state. Using the Lie algebra difference operator \(a \boxminus b \triangleq \log_{\text{SE}(3)}(a \cdot b^{-1})\), the framework imposes a geodesic temporal consistency loss: $\(\mathcal{L}_{\text{temp}} = \sum_{t=2}^{T} \left( \|\hat{p}_t^{\text{hand}} \boxminus \Phi_{\text{hand}}(p_{t-1}, z_t)\| + \beta \|\hat{p}_t^{\text{obj}} \boxminus \Phi_{\text{obj}}(p_{t-1}, z_t)\| \right)\)$ By applying spectral normalization to maintain an \(L\)-Lipschitz contraction factor \(L < 1\) (\(L=0.95\) empirically), Proposition 2 establishes that the cumulative tracking error remains uniformly bounded over arbitrary sequence lengths: \(\sup_{t \in [1, T]} \mathbb{E}[d_{\mathcal{M}}(\hat{p}_t, p_t^*)] \le \frac{\epsilon}{1 - L}\). This effectively acts as a geodesic low-pass filter, preventing trajectory divergence during extended manipulation horizons.

3. MDP Pose Sufficiency and Cross-Embodiment Co-Training: Unlocking Internet Video for Control Cross-embodiment transfer is traditionally hampered by visual domain gaps between human hands and robot end-effectors. Under a Markov Decision Process (MDP) formulation for quasi-static manipulation, task rewards depend purely on the relative transformation \(\delta_t = (p_t^{\text{obj}})^{-1} \cdot p_t^{\text{hand}}\). Extraneous factors like lighting, background clutter, and visual morphology are completely redundant. By projecting both the translated human demonstration dataset \(\mathcal{D}_{\text{human}}^{\text{trans}}\) and teleoperated robot demonstrations \(\mathcal{D}_{\text{robot}}\) into a unified 6-DoF displacement action space, policy \(\pi_\theta(a_t \mid p_{t-H:t})\) is co-trained via joint behavioral cloning: $\(\mathcal{L}_{\text{policy}}(\theta) = \mathbb{E}_{(p,a)\sim\mathcal{D}_{\text{robot}}} [\ell(\pi_\theta(p), a)] + \mu \mathbb{E}_{(\hat{p},\hat{a})\sim\mathcal{D}_{\text{human}}^{\text{trans}}} [\ell(\pi_\theta(\hat{p}), \hat{a})]\)$ With \(\mu=0.3\) balanced via initial gradient norms, thousands of translated human interaction clips seamlessly augment scarce robot teleoperation data, enhancing generalizability across complex bimanual workflows.

Loss & Training

The framework follows a two-stage training strategy: 1. Pose Translation Diffusion: Trained on OakInk V2, H2O, AssemblyHands, and Assembly101 for 200 epochs using 8 A100 GPUs, AdamW optimizer (learning rate \(3 \times 10^{-4}\), cosine annealing, 5K warmup steps), with total objective \(\mathcal{L}_{\text{diff}} + \alpha \mathcal{L}_{\text{temp}}\) (\(\beta=0.5\)). The score network comprises a 12-layer Transformer (hidden dimension 512, 8 heads). The motion prior \(\Phi\) is a 2-layer GRU with spectral normalization pretrained for 20 epochs. 2. Policy Co-Training: Evaluated on the Ego Humanoid Manipulation Benchmark in NVIDIA Isaac Lab with a Unitree H1 robot. Action Transformer models ingest a history of \(H\) pose observations to predict future action trajectories, incorporating converted TACO and Mimicplay human demonstrations.

Key Experimental Results

Main Results

The method is evaluated on hand-object pose translation accuracy on H2O (comparing against poses extracted from pixel-synthesis models via HAMER and FoundationPose) and on policy task execution on the Unitree H1 humanoid benchmark.

Table 1: Exo-to-Ego Hand Pose Translation Accuracy on H2O (Selected from Table 1 in paper)

Method Unseen Objects PA-MPJPE (mm)↓ Unseen Objects F@5mm↑ Unseen Objects F@15mm↑ Unseen Actions PA-MPJPE (mm)↓ Unseen Actions F@15mm↑
Pix2PixHD 42.6 0.142 0.412 46.3 0.374
Exo2Ego-V 24.1 0.312 0.645 28.4 0.601
EgoWorld 19.4 0.421 0.762 23.1 0.714
4Diff 18.2 0.448 0.782 21.7 0.738
EgoX 15.7 0.521 0.844 18.9 0.798
Ours (Exo2EgoPolicy) 11.4 0.685 0.912 14.2 0.874

Table 2: Policy Performance on Long-Horizon Manipulation Tasks (Selected from Table 4 in paper)

Method Category & Representative Baseline Seen Mean Success Rate SR (%)↑ Seen Mean Progress Rate PSR (%)↑ Unseen Mean Success Rate SR (%)↑ Unseen Mean Progress Rate PSR (%)↑
Without human data (ACT) 2.2 26.5 0.6 23.5
Without human data (EgoVLA-NoPretrain) 26.7 54.9 11.2 36.2
Human data without view translation (Exo-only) 27.6 56.3 12.1 37.6
Pixel-space view translation (EgoX) 41.5 73.1 23.9 61.9
Egocentric human pretraining (EgoVLA) 45.9 80.8 28.8 69.1
Pose-space view translation (Ours) 53.3 84.3 36.1 74.5

Ablation Study

Ablations on H2O (Unseen Objects pose translation) and the Isaac Lab benchmark (Unseen policy configuration) demonstrate the necessity of each architectural component.

Table 3: Ablation Analysis of Key Components (Table 5 in paper)

Configuration PA-MPJPE (mm)↓ F@15mm↑ Short-Horizon SR (%)↑ Long-Horizon SR (%)↑ Note
Ours (Full Model) 11.4 0.912 73.7 36.1 Full proposed pipeline
(a) w/o object-relative formulation 18.4 0.712 58.3 20.1 World-coordinate prediction suffers from extrinsics ambiguity
(b) w/o geometric latent \(z_t\) 21.6 0.671 54.7 17.4 Direct regression fails to resolve underdetermined projection
(c) w/o autoregressive state \(p_{t-1}\) 17.1 0.783 62.4 24.8 Lacks temporal conditioning in score network
(d) w/o temporal consistency loss \(\mathcal{L}_{\text{temp}}\) 19.3 0.748 57.9 19.8 Missing Lie-algebra motion filter causes severe policy jitter

Key Findings

  • Raw exocentric data provides marginal utility: Training policies directly on exocentric poses without viewpoint transformation (Exo-only) improves Seen short-horizon SR by only 1.1% over the robot-only baseline (64.6% to 65.7%), demonstrating that geometric perspective misalignment prevents meaningful knowledge transfer.
  • Pose abstraction surpasses pixel-level hallucination: Compared to leading generative video models like EgoX, directly inferring SE(3) pose cuts hand joint estimation error from 15.7mm to 11.4mm on H2O and improves Unseen long-horizon policy success rate from 23.9% to 36.1% (+12.2% absolute gain), proving that visual synthesis artifacts severely degrade policy execution.
  • Temporal regularization is decisive for multi-stage skills: Discarding the temporal loss \(\mathcal{L}_{\text{temp}}\) slashes long-horizon manipulation success rate from 36.1% down to 19.8%, highlighting that contractive geodesic regularization is essential to suppress cascading kinematic errors across subtask transitions.

Highlights & Insights

  • Conceptual Paradigm Shift: Replaces computationally heavy and artifact-prone pixel-level video translation with structured geometric pose inference on the SE(3) manifold, aligning representations directly with control requirements.
  • Extrinsics Invariance via Local Anchoring: By defining hand poses relative to the target object coordinate frame, the method cleanly marginalizes out uncalibrated camera motion without requiring hardware instrumentation.
  • Contractive Error Guarantee: Integrates Lie algebra kinematics with a spectrally normalized transition prior, providing formal non-divergence error bounds that resolve generative high-frequency trajectory jitter.

Limitations & Future Work

  • Reliance on Upstream Perception Quality: The framework assumes that exocentric hand and object poses can be reliably initialized via off-the-shelf estimators (HAMER, FoundationPose); severe visual occlusions or untextured objects in third-person video may degrade input latent conditioning.
  • Quasi-Static Rigidity Assumptions: The MDP sufficiency argument relies on relative rigid-body geometry, which may face representational limits when handling deformable objects, fluids, or complex topological changes.
  • Future Directions: Extending the framework to incorporate whole-body mobile manipulation dynamics and closed-loop tactile force feedback.
  • vs EgoWorld / 4Diff / EgoX: While prior works focus on generating photorealistic egocentric video frames, they suffer from high-frequency spatial distortion and temporal flickering; Exo2EgoPolicy operates directly on kinematic poses, yielding smooth, control-ready trajectories.
  • vs EgoVLA: EgoVLA pretrains exclusively on first-person human data, which remains scarce; Exo2EgoPolicy unlocks internet-scale exocentric videos via geometric pose translation, outperforming EgoVLA on challenging long-horizon benchmarks.
  • vs DexCap / HumanPlus: Avoids specialized tracking gloves, exoskeletons, or hardware retargeting, transferring manipulation knowledge directly from monocular RGB video into humanoid robot control.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulating cross-view video imitation as SE(3) pose diffusion with contractive temporal filtering is an elegant and theoretically grounded departure from pixel synthesis.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluated across pose translation benchmarks (H2O, Assembly101) and 12 humanoid manipulation tasks in Isaac Lab with comprehensive ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulation, solid theoretical justification via MDP analysis, and crisp structural organization.
  • Value: ⭐⭐⭐⭐⭐ Establishes a highly practical and scalable foundation for converting abundant web-scale human videos into robotic manipulation policies.