Lifting Ego World Models for Planning and Control

Conference: ECCV 2026
Paper: ECCV Official Page
Code: https://alexn.wang/lwm
Area: Robotics & Embodied AI
Keywords: Embodied World Models, Action Space Lifting, Waypoint Planning, Egocentric Vision, Cross-Entropy Method (CEM)

TL;DR

Addressing the challenge that high-dimensional joint action spaces make human-like world models intractable for search-based planning, this paper introduces a lightweight policy that maps sparse 2D leaf joint waypoints on the current frame into joint action sequences, lifting frozen world models into an 8D action space that reduces CEM planning error by 3.8× at substantially lower compute cost.

Background & Motivation

World models offer a powerful paradigm for embodied intelligence by predicting future environmental evolution conditioned on the agent's actions, enabling physical common-sense learning and prospective simulation. As research scales from simplified 2D grids and discrete navigation toward rich human-like embodiments, the controlling action spaces have grown prohibitively high-dimensional. For example, controlling an articulated human-like body modeled via full-body motion capture (such as XSens) requires specifying continuous angular displacements across dozens of joints (e.g., 48 dimensions per step) to coordinate reaching, bending, and locomotion. This high dimensionality causes search-based trajectory optimization methods like the Cross-Entropy Method (CEM) to suffer severely from the curse of dimensionality, scaling exponentially with both sequence length and action dimension and often degenerating or stalling in local minima.

Prior approaches for embodied navigation and manipulation often condition policies or planners on goal observations. However, in egocentric video, the agent's own body is mostly out of frame; consequently, a goal image provides scarce geometric clues regarding target joint configurations and body poses. Furthermore, goal observations reside in high-dimensional image space, making them ill-suited for gradient-free sampling or search. Directly searching over raw continuous joint angles frequently produces unnatural, physically uncoordinated motions, failing to solve hybrid tasks that require both navigation and precise object interaction.

Inspired by mammalian visuomotor control—where visuospatial goal representations in the posterior parietal cortex are transformed into joint-level motor commands in the motor cortex—this paper proposes a principled abstraction lifting strategy. Rather than planning directly within unconstrained high-dimensional joint spaces, the search space is lifted to an intuitive, low-dimensional visual goal representation defined directly in the agent's first-person view. Core idea: train a lightweight diffusion policy that translates sparse 2D leaf-joint waypoints on the current frame into low-level whole-body joint action sequences, composing it with a frozen world model into a Lifted World Model (LWM) so that CEM planning can be performed directly and efficiently in an 8-dimensional waypoint space.

Method

Overall Architecture

The core architecture of the Lifted World Model (LWM) leaves the underlying high-fidelity low-level world model (PEVA) completely frozen, bridging the gap between high-level visual intent and low-level physical execution via a lightweight goal-conditioned diffusion policy network \(\pi_\theta\). The overall framework consists of three hierarchical stages: 1. High-Level Action Space Formulation: A sparse set of 2D waypoints \(a^\text{HL}\) is defined on the current egocentric observation frame \(o_t\), specifying target positions in the image plane for key leaf joints; 2. Action Lifting Policy: The policy network \(\pi_\theta\) consumes past observation frames, historical pose sequences, and the high-level 2D waypoints to generate a sequence of \(T\) low-level joint displacements \(a^\text{LL}_{t:t+T-1}\); 3. Autoregressive World Model Rollout & Planning: The frozen low-level world model \(f_\phi\) unrolls this joint action sequence autoregressively to generate future predicted observations \(o_{t+1:t+T}\), allowing a high-level Cross-Entropy Method (CEM) planner to evaluate perceptual similarity against the goal and iteratively refine the waypoint sampling distribution.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Current Observation & Pose History<br/>o_t, p_t"] --> B["2D Leaf Joint Waypoints<br/>Pelvis / Head / Hands 8D Action Space"]
    B --> C["High-Level Conditioned Policy<br/>Lightweight Diffusion Policy"]
    C --> D["Low-Level Joint Action Sequence<br/>T-Step 48D Articulated Displacements"]
    D --> E["Frozen Low-Level World Model<br/>Autoregressive Future Simulation"]
    E --> F["Cross-Entropy Method Planning<br/>Iterative Search in 2D Waypoint Space"]

Key Designs

1. 2D Leaf Joint Waypoint Space: Eliminating High-Dimensional Search Explosion In human-like embodiments, whole-body motion involves complex parent-child kinematic dependencies and angular redundancies across dozens of degrees of freedom. Rather than relying on dense 3D trajectories or cumbersome 6D poses, the authors designate four critical leaf joints from the XSens kinematic tree: the pelvis, head, left hand, and right hand. During training, forward kinematics transforms the 3D joint positions of the ground-truth goal pose \(p_g\), which are then projected via the camera projection matrix \(P\) onto the current image plane \(o_t\), resulting in a compact 8-dimensional high-level action: $\(a^\text{HL} = \{w_\text{pelvis}, w_\text{head}, w_\text{left\_hand}, w_\text{right\_hand}\} \in [0, 1]^8\)$ Expressing goals as 2D image coordinates removes the requirement for explicit metric depth sensors, grounds goals directly in the visible geometric context of the scene, and shrinks the per-step planning search space from 48 dimensions down to just 8.

2. Waypoint Masking Mechanism: Enabling Sparse and Counterfactual Intent In practical robotics tasks, user instructions or intermediate goals frequently specify only a subset of joints—such as commanding only the pelvis for locomotion, or commanding only the hands for table-top manipulation—while other joints may naturally fall outside the field of view. To equip the policy with the capability to hallucinate plausible and coordinated poses for unspecified joints, the training regimen incorporates independent waypoint masking. Half of the training batches retain full waypoint annotations, while in the remaining half, each joint waypoint is independently dropped with probability 0.5; when all waypoints are masked, the model gracefully falls back to the unconditioned motion prior \(p(a^\text{LL}_{t:t+T-1}|o_t, p_t)\). This regularizer compels the policy to capture human whole-body coordination priors, allowing it at inference time to generate balanced, natural postures even when given only a single sparse waypoint.

3. Cross-Entropy Planning on the Lifted World Model: Low-Dimensional Search for High-Dimensional Execution Directly running CEM over low-level joint actions leads to optimization stagnation and unnatural, disjointed postures due to the non-convex, unconstrained parameter space. Under the Lifted World Model, CEM optimization is transferred entirely into the 2D waypoint space. In each planning iteration, \(N\) candidate waypoint sets are sampled from a low-dimensional prior \(p_\text{HL}(\cdot)\), mapped through policy \(\pi_\theta\) into physically coherent joint action sequences, and unrolled through world model \(f_\phi\) to yield predicted final observations \(\hat{o}_g\). By ranking the perceptual distance cost \(C(\hat{o}_g, o_g)\) against the target image, the \(M\) elite candidates are selected to update the Gaussian sampling distribution. Decoupling high-level intention search from low-level motion synthesis prevents kinematic breakdown and accelerates planning convergence.

Loss & Training

The underlying PEVA world model remains completely frozen throughout training. The policy \(\pi_\theta\) builds upon a Diffusion Policy architecture enhanced with a DINOv3 visual encoder, later spatial pooling, and 3D positional embeddings. The network is optimized via standard conditional diffusion mean-squared error on Gaussian noise \(\epsilon \sim \mathcal{N}(0, I)\): $\(\mathcal{L}(\theta) = \mathbb{E}_{k, \epsilon, a^\text{LL}, o_t, p_t, a^\text{HL}}\left[ \|\epsilon - \epsilon_\theta(a^{\text{LL}, k}, k, \text{Encoder}(o_t, a^\text{HL}), p_t)\|^2 \right]\)$ where \(k\) denotes the diffusion timestep. Waypoints are visually rendered onto the input frame \(o_t\) as colored discs corresponding to specific joints, allowing the visual backbone to directly attend to spatial relationships between target joints and scene affordances.

Key Experimental Results

Main Results

On 128 hybrid navigation and interaction planning tasks constructed from the Nymeria dataset, search-based planning in the lifted waypoint space (Lifted CEM) was compared against direct joint-space search (PEVA CEM) and various goal-conditioned policy baselines. Performance is evaluated using Mean Joint Error (MJE) in meters between the planned final pose and the ground-truth target pose across leaf joints, intermediate joints, and all joints.

Method	Leaf MJE (m) ↓	Int. MJE (m) ↓	All MJE (m) ↓	Note
Initial Distance	0.724	0.697	0.704	Starting pose baseline error
Copy Baseline (1-NN)	0.696	0.663	0.672	Nearest-neighbor retrieval
Unconditioned Policy	0.677	0.641	0.650	Open-loop unconditioned prior
Diffusion Policy	0.605	0.578	0.585	Image goal conditioned baseline
Hierarchical Diffusion Policy (HDP)	0.540	0.502	0.512	Hierarchical baseline
PEVA CEM (Low-Level Joint Search)	0.637	0.608	0.616	6 iterations / 64 samples
Lifted CEM (3D Waypoints)	0.453	0.407	0.420	Waypoints with depth values
Lifted CEM (2D Waypoints, Ours)	0.411	0.359	0.374	33.0cm reduction; 3.8× over PEVA CEM

Ablation Study

On the Nymeria validation set, the architectural enhancements and conditioning modalities of the policy were ablated across both unconditioned and goal-conditioned generation settings.

Model Configuration	Uncond. Leaf MJE (m) ↓	Uncond. All MJE (m) ↓	Goal Cond. Leaf MJE (m) ↓	Goal Cond. All MJE (m) ↓	Key Contribution
Initial Distance	0.445	0.426	0.445	0.426	Initial error reference
Random Weights	0.749	0.718	0.735	0.703	Untrained network
Base Policy	0.427	0.405	0.414	0.392	Standard NoMaD architecture
+ Architecture Changes	0.406	0.384	0.388	0.367	DINOv3 + spatial pooling
+ Pose Context History	0.359	0.337	0.343	0.323	Proprioceptive state context
+ Waypoint Conditioning	0.353	0.331	0.262	0.243	Large gain in goal steering
+ Waypoint Masking (Final)	0.437	0.415	0.243	0.226	Optimal goal alignment
3D Waypoint Conditioning	0.360	0.338	0.243	0.226	Added depth information
3D Waypoints + Masking	0.415	0.398	0.223	0.208	Minimal gain in policy; harder search

Key Findings

• Low-level search degrades rapidly: Direct CEM optimization over PEVA's joint action space reduces all-joint MJE by only 8.8cm (from 0.704m to 0.616m), often diverging after the very first iteration. In contrast, Lifted CEM cuts error by 33.0cm down to 0.374m, representing a 3.8× larger improvement over the naive search baseline.
• 2D outperforms 3D in search-based planning: Although augmenting waypoints with ground-truth depth marginally improves isolated policy fitting (0.208m vs 0.226m), it expands the action dimension from 8 to 12. During CEM planning, this increased search volume hampers convergence, causing 3D Lifted CEM (0.420m) to fall behind 2D Lifted CEM (0.374m).
• Zero-shot generalization across unseen environments: When the policy is trained strictly on data excluding Nymeria locations 6, 19, and 34, Lifted CEM achieves 0.362m all-joint MJE on tasks in those held-out environments—vastly outperforming PEVA CEM (0.553m) and trailing the unconstrained policy (0.333m) only slightly, demonstrating exceptional structural generalization.

Highlights & Insights

• Decoupling visuospatial intent from kinematic execution: By identifying that egocentric goal images fail because the agent's body is invisible, the paper elegantly shifts goal-conditioning to projected 2D waypoints, mimicking human motor hierarchy and avoiding unnatural joint hallucinations.
• Non-invasive, plug-and-play lifting: The architecture leaves the heavyweight, compute-intensive world model completely untouched, showing that a modular, lightweight adapter policy can effectively lift model abstraction without costly retraining.
• Scene-aware kinematic context: Given identical hand waypoints, the policy synthesizes a grasping gesture when facing a stove, but a relaxed forward stride when facing an open room, demonstrating that it grounds abstract waypoints in visual affordances.

Limitations & Future Work

• Fixed-horizon action chunking: The policy operates with a fixed horizon of \(T=8\) steps, requiring heuristic chaining for long-horizon planning rather than adaptive multi-scale temporal abstractions.
• Field-of-view dependency: When the target goal lies completely behind the agent, 2D image projections become undefined or degenerate; incorporating panoramic views or topological memory graphs would enhance full-turnaround planning.
• Absence of fine-grained dexterous manipulation: The current waypoint space terminates at the wrists and palm centers, leaving multi-finger articulated contact dynamics unmodeled.

Related Work & Insights

• vs PEVA / DexWM: While PEVA unlocked full-body conditioned egocentric video prediction, its 48D action space rendered it unplannable; this work lifts PEVA into a practical model-predictive controller.
• vs NoMaD / Diffusion Policy: Standard diffusion policies condition on goal images, which fail in first-person human embodiment due to self-occlusion; 2D waypoints provide a far more robust conditioning signal.
• vs MDM / OmniControl: Human motion synthesis typically relies on dense 3D keypoint tracks; this work proves that sparse 2D endpoints integrated into a closed-loop world model suffice for goal-directed locomotion and interaction.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant abstraction of complex humanoid action spaces into 2D leaf joint waypoints, transforming an intractable world model into an efficient planner.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations covering masking, visibility splits, 2D vs 3D, compute budget curves, and unseen environment transfer.
• Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative, sound theoretical motivation grounded in neuroscience, and clear visual illustrations.
• Value: ⭐⭐⭐⭐⭐ Provides a highly practical blueprint for controlling and planning with foundation world models on complex high-dimensional embodiments.