EgoSim: Egocentric World Simulator for Embodiment Interaction Generation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://egosimulator.github.io
Area: 3D Vision
Keywords: Egocentric World Simulator, Embodied Interaction Generation, 3D Point Cloud State Updating, Diffusion Transformer, Cross-Embodiment Transfer

TL;DR¶

EgoSim introduces a closed-loop egocentric world simulator that couples a geometry-action-aware video diffusion model with an updatable 3D point cloud world state, generating spatially consistent, continuous interaction videos with seamless cross-embodiment transfer.

Background & Motivation¶

Embodied intelligence and interactive world simulators represent foundational technologies for spatial computing, virtual reality, and robotic policy learning. Propelled by recent advances in Diffusion Transformers (DiT), generative world modeling has progressed from early latent-space transition dynamics to high-fidelity visual synthesis. Nevertheless, mainstream world simulators (e.g., Genie, The Matrix) predominantly target third-person panoramic exploration or coarse directional navigation, leaving the observer as a passive spectator. In contrast, in egocentric settings, humans and robots act as active participants executing highly dexterous manipulation, which is accompanied by aggressive egocentric camera motions and demands precise, fine-grained physical reactions to end-effector interactions under strict 3D physical constraints.

Recent egocentric simulators (such as PlayerOne, Hand2World, and DWM) have explored generating first-person human-object interactions, but remain bottlenecked by two critical limitations. First is the lack of persistent 3D spatial consistency and state persistence: video models relying solely on implicit camera pose injection suffer from structural collapse and severe viewpoint drift under large viewpoint shifts; furthermore, even architectures that explicitly decouple static backgrounds treat the physical scene as a "frozen" environment, failing to update the underlying 3D world state across multi-stage interactions and causing manipulated objects to unrealistically snap back to their initial configurations. Second is the severe constraint on data scalability: existing methods depend heavily on rigidly calibrated multi-view motion capture rigs or synthetic simulation environments, which are orders of magnitude smaller than open-world monocular egocentric videos, severely restricting generalization to in-the-wild interactions.

To overcome geometric inconsistency, state amnesia, and training data bottlenecks, this paper proposes EgoSim, an end-to-end closed-loop egocentric world simulator. EgoSim models the 3D environment as an evolving, editable point cloud memory and introduces an automated pipeline to extract geometrically aligned scene-interaction quadruplets from uncurated monocular videos. Core Idea: model continuous egocentric simulation as a closed-loop cycle alternating between "geometry-action-aware visual observation synthesis conditioned on rendered 3D scenes and projected keypoints" and "interaction-aware 3D state updating via open-vocabulary tracking and incremental TSDF fusion," ensuring strict spatial consistency and seamless cross-embodiment generalization.

Method¶

Overall Architecture¶

EgoSim operates via a cyclic closed-loop pipeline alternating between visual observation generation and persistent 3D state updating. At stage $k$, the simulator maintains an explicit 3D world state $S_{k-1}$ stored as a point cloud. The input action $A_k = (C_k, H_k)$ is explicitly decoupled into a 6-DoF camera trajectory $C_k$ and an embodiment interaction sequence $H_k$.

The simulation step comprises two sequential operations: first, the Geometry-action-aware Observation Simulation model renders the static background projection $\Pi(S_{k-1}; C_k)$ along camera path $C_k$ and synthesizes the action-induced dynamic residual $\Delta O(H_k)$, yielding visual observation $O_k$: $$O_k = \Pi(S_{k-1}; C_k) + \Delta O(H_k)$$ Subsequently, the Interaction-aware State Updating module $\mathcal{U}$ extracts the updated physical layout of manipulated objects from observation $O_k$ and incrementally fuses them back into the 3D point cloud, producing the updated world state $S_k$: $$S_k = \mathcal{U}(S_{k-1}, O_k)$$ This persistent state updating prevents manipulated objects (e.g., an opened door or an extracted item) from reverting to their original layouts in subsequent stages, providing a physically grounded geometric anchor for long-horizon simulation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Initial 3D Scene State S_k-1<br/>and Action Sequence (C_k, H_k)"] --> B["Geometry-Action-Aware Observation Simulation<br/>DiT combines rendered point cloud and keypoints to generate O_k"]
    B --> C["Interaction-Aware Object State Decoupling & Update<br/>SAM3 tracking and hierarchical filtering extract latest object geometry"]
    C --> D["Pose Alignment & Incremental TSDF Fusion<br/>Sim3 alignment and TSDF point cloud fusion update S_k"]
    D -->|Updated 3D state closed-loop transition| A

Key Designs¶

1. Geometry-Action-Aware Observation Simulation: Decoupling Static Scene Rendering from Cross-Embodiment Action Representation

To prevent background distortion under drastic camera motions, this design projects the 3D point cloud along camera trajectory $C_k$ to construct a visual reference video, providing an explicit spatial anchor for video diffusion. For embodiment actions $H_k$, rather than conditioning on dense, embodiment-specific hand meshes, the framework extracts core 3D joint keypoints and perspective-projects them onto the 2D image plane, forming an action keypoint video $O_{action}$. This perspective projection natively captures depth-dependent foreshortening while serving as a universal, embodiment-agnostic action proxy that bridges human hands and robotic grippers. To handle unobserved view frustums and scanning occlusions in the rendered point cloud, a binary visibility mask video $M$ explicitly marks unrendered regions. The diffusion backbone, instantiated as a Diffusion Transformer (DiT), operates in the latent space of a pretrained video VAE with input channels expanded to 52, concatenating noisy latent $z_t$, background reference latent $z_{bg}$, action latent $z_{hand}$, and downsampled mask $M$: $$z_{in}^{(t)} = \text{Concat}(z_t, z_{bg}, z_{hand}, M)$$ Initializing the DiT with video inpainting weights provides a dual generative prior: it functions as a generative inpainting model over unobserved regions ($M=1$) while acting as an identity mapping on rendered backgrounds ($M=0$), focusing model capacity purely on synthesizing action-induced interactive dynamics.

2. Interaction-Aware Object State Decoupling & Update: Open-Vocabulary Tracking and Hierarchical Geometric Filtering

Directly reconstructing point clouds from generated interaction videos introduces severe ghosting artifacts and floating noise caused by human hands and non-rigid skin deformations. To maintain a clean, persistent 3D world state, this design proposes a training-free interaction-aware decoupling mechanism. A vision-language model (VLM) first parses interaction-related object phrases (e.g., "cup lid", "ladle") from text cues. An open-vocabulary segmentation and tracking pipeline based on SAM3 then predicts 3D point masks $\mathcal{M}_{int}^i$ for each interactive object. Hierarchical filtering suppresses false positives by anchoring object proposals within the spatial vicinity of the hand and validating spatiotemporal IoU overlap alongside multi-view depth consistency. During point reconstruction, the static background excluding $\mathcal{M}_{int}$ is fused into $\mathcal{P}_{bg}$, while interactive objects are tracked across frames and only their latest geometry from the final frame is composited into the scene, yielding candidate state $\hat{S}_k$. This decoupling accommodates articulated rotations (e.g., cabinet doors) and multi-part assembly tasks without dynamic contamination.

3. Pose Alignment & Incremental TSDF Fusion: Long-Horizon Spatiotemporal Consistency via Overlapping Pose Registration

In multi-stage continuous simulation, monocular depth estimation and trajectory tracking inevitably suffer from scale ambiguity and cumulative drift; naive concatenations of per-stage point clouds result in geometric fracture and double-image artifacts. To resolve this, the incremental state fusion module aligns consecutive states using the Umeyama algorithm to compute a 3D similarity transformation $\text{Sim}(3)$ (solving optimal rotation, translation, and global scale) across overlapping camera poses $C_{k-1}$ and $C_k$. Once aligned to the global reference frame, Truncated Signed Distance Field (TSDF) fusion performs volumetric space-weighted averaging over overlapping static regions, eliminating single-frame depth noise and seamlessly expanding observed geometry. Concurrently, point clusters belonging to interactive objects are updated using the latest observation. This mechanism guarantees that the static environment expands stably while maintaining an immutable global world coordinate system for long-horizon simulation.

A Worked Example: Ice Scooping and Continuous Interaction¶

Consider the two-stage "Scooping ice from a container into a cup" task from the EgoDex dataset: - Stage 1 Input & Rendering: Given the initial frame, human hands are inpainted via SAM3 and Qwen-Image-Editing, and DepthAnything3 backprojects the image into initial point cloud $S_0$. The user specifies camera pitch trajectory $C_1$ and scooping action $H_1$. - Stage 1 Observation Synthesis: Point cloud $S_0$ rendered along $C_1$ supplies the 3D tabletop background; 3D hand joints are projected into 2D keypoints. DiT processes the 52-channel input to generate a 61-frame video $O_1$ showing the spoon entering the ice box and scooping an ice cube. - Stage 1 State Persistence: The VLM identifies "spoon" and "ice cube"; SAM3 tracks their trajectories while isolating hand points. Hierarchical filtering registers the ice cube in its elevated 3D position at frame 61. TSDF fusion integrates this update into world state $S_1$. - Stage 2 Continuous Simulation: In the second stage (generating a 121-frame sequence in total), the simulator receives updated state $S_1$ and action $H_2$ (moving the spoon over the glass). Rendered from $S_1$, the scene reflects the ice cube already displaced from the box. DiT synthesizes the drop into the glass, completely preventing the ice cube from snapping back to its original location.

Loss & Training¶

Backbone Architecture: Initialized from Wan-2.1-Fun-14B-InP, with DiT input channels expanded to 52 to ingest noisy latents, rendered background latents, action keypoint latents, and binary masks. Text encoder (T5), visual VAE, and CLIP image encoders remain frozen.
Video Resolution & Timing: Generates 61-frame clips at $832 \times 480$ resolution and 16 FPS.
Optimization Strategy: End-to-end full parameter fine-tuning of the DiT using Flow Matching. Optimized via AdamW with learning rate $1 \times 10^{-5}$ and per-GPU batch size 4 across 8 NVIDIA H200 GPUs for 4,000 steps. Inference employs a Flow Matching ODE solver with 50 steps and classifier-free guidance (CFG) scale of 1.0.
Data Pipeline: Trained on 400K aligned scene-interaction quadruplets from EgoDex (240K clips) and EgoVid (160K clips), and fine-tuned on AgiBot-World (50K clips) for robotics. The lightweight EgoCap mobile capture pipeline enables uncalibrated 3DGS scanning and 6-DoF relocalization for rapid real-world adaptation.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted under the single-clip setting across 100 randomly sampled unseen test videos from EgoDex (fine-grained tabletop manipulation) and EgoVid (in-the-wild interactions). Structural consistency is measured via Depth-ERR (depth discrepancy against ground truth), and viewpoint consistency is measured via Cam-ERR (Plücker ray error).

Dataset	Method	PSNR ↑	SSIM ↑	LPIPS ↓	Depth-ERR ↓	Cam-ERR ↓
EgoDex (Tabletop Scene)	Wan-2.1-14B-InP	17.998	0.447	0.708	42.335	0.0300
EgoDex (Tabletop Scene)	Mask2IV	20.622	0.814	0.299	38.339	0.0181
EgoDex (Tabletop Scene)	CosHand	17.119	0.776	0.386	56.524	0.0499
EgoDex (Tabletop Scene)	InterDyn	22.250	0.830	0.255	44.345	0.0226
EgoDex (Tabletop Scene)	EgoSim (Ours)	25.056	0.896	0.170	8.888	0.0013
EgoVid (In-the-wild)	Wan-2.1-14B-InP	11.754	0.430	0.503	34.470	0.0174
EgoVid (In-the-wild)	Mask2IV	12.311	0.414	0.571	34.413	0.0175
EgoVid (In-the-wild)	CosHand	11.805	0.408	0.600	39.373	0.0516
EgoVid (In-the-wild)	InterDyn	14.612	0.466	0.484	38.180	0.0308
EgoVid (In-the-wild)	EgoSim (Ours)	16.684	0.509	0.421	19.260	0.0105

Ablation Study¶

Ablations validate core design choices on EgoDex (Table 3), multi-stage continuous rollouts (Table 2), and robotic cross-embodiment pretraining (Table 4).

Config / Setting	PSNR ↑	SSIM ↑	LPIPS ↓	Depth-ERR ↓	Cam-ERR ↓	Note
EgoDex Ablation: Full model	25.056	0.896	0.170	8.888	0.0013	Full pipeline with true trajectory rendering and mask guidance
EgoDex Ablation: w/o trajectory	23.380	0.845	0.244	10.238	0.0015	Static frame replicated; lacks camera motion cues, leading to distortion
EgoDex Ablation: w/o mask	23.988	0.886	0.186	14.124	0.0022	All-black mask input; relies solely on internal generative prior
Continuous Generation: Single-clip (61 frames)	25.056	0.896	0.170	8.888	0.0013	Ground-truth first frame and initial ground-truth point cloud
Continuous Generation: Closed-loop (121 frames)	19.165	0.835	0.220	10.943	0.0017	Evaluated across two stages; 3D state updated from generated observations
Robotics Transfer: w/o hand pretrain	16.36	0.69	0.31	-	-	Trained on AgiBot robot data for 1,400 steps
Robotics Transfer: w/ hand pretrain	18.67	0.72	0.28	-	-	Pretrained on egocentric hand data (1,200 steps), finetuned on robot (200 steps)

Key Findings¶

Explicit 3D geometric grounding dictates spatial control fidelity: Compared to the 2D mask-conditioned InterDyn, EgoSim reduces Depth-ERR from 44.345 to 8.888 and suppresses Cam-ERR by over an order of magnitude (from 0.0226 to 0.0013). Removing camera trajectory rendering (w/o trajectory) drops PSNR by 1.676 dB, demonstrating that projecting 3D point clouds along actual viewpoints is essential to eliminate parallax hallucination.
State updating effectively bounds long-horizon error accumulation: In the 121-frame continuous generation rollout where only the initial frame is provided, EgoSim (Continuous) retains a competitive PSNR of 19.165 and SSIM of 0.835, with Depth-ERR increasing moderately by only 2.055. This confirms that decoupling interactive objects and TSDF fusion prevents floating artifacts from destabilizing future stages.
Dexterous interaction priors transfer across embodiments: With only 200 fine-tuning steps on AgiBot robot data, the model pretrained on human hand data significantly outperforms the model trained exclusively on robot data for 1,400 steps (+2.31 dB PSNR, +0.03 SSIM), proving that projected keypoints and 3D background interaction dynamics possess cross-embodiment physical universality.

Highlights & Insights¶

Unified 3D keypoint projection as a cross-embodiment interface: Projecting 3D keypoints into the 2D camera view eliminates the topological rigidity of dense hand meshes, naturally models geometric perspective foreshortening, and allows seamless re-targeting to robotic grippers via forward kinematics.
Synergy between inpainting priors and explicit geometry: Initializing the DiT with video inpainting weights enables it to act as an identity function preserving rendered 3D background points while serving as a generative inpainting prior that cleanly fills occluded regions and synthesizes dynamic hand-object contacts.
Training-free, interaction-aware closed-loop state maintenance: Combining VLM semantic grounding, SAM3 instance tracking, and TSDF fusion achieves persistent 3D world state updates without training complex dynamic reconstruction networks, closing the generation-to-state-update loop.

Limitations & Future Work¶

Vulnerability to monocular depth and pose tracking failures: In severe occlusions, mirror reflections, or rapid jerky motions, monocular depth estimation and SLAM can suffer scale drift and voids, leading to distorted initial point clouds.
Discrete point representations struggle with continuous topological changes: While point clouds and TSDF excel at rigid translation and articulated joint rotation, modeling non-topological deformations (e.g., kneading dough, fluid splashing) remains difficult in discrete point representations.
Future Directions: Integrating 3D Gaussian Splatting (3DGS) or neural fields as continuous explicit representations, alongside physics-based contact constraints and multi-view geometric priors to enhance tactile and collision realism.

vs Wan-2.1-14B-InP: Wan 2.1 operates purely on 2D pixel inpainting without 3D spatial awareness or explicit action conditioning; EgoSim introduces 3D point cloud projections and action keypoints to transform video inpainting into an interactive world model.
vs InterDyn: InterDyn relies on 2D hand masks and ControlNet injection, frequently conflating hand motion with camera motion and causing severe background drifting; EgoSim grounds generation on explicit 3D point cloud projections.
vs DWM (Dexterous World Models): DWM decouples static backgrounds from dynamic actions using rendered point maps, but treats the 3D scene as static and unchangeable; EgoSim introduces an interaction-aware state update module to persistently update the environment across multi-stage rollouts.
vs VIPE / Spatia: VIPE and related methods primarily focus on filtering dynamic objects to reconstruct clean static scenes; EgoSim goes further by tracking manipulated objects and incrementally fusing their updated poses back into the 3D scene state.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First closed-loop egocentric simulator integrating updatable 3D world states with video diffusion and scalable monocular data extraction.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 400K-scale datasets, single-clip baselines, continuous multi-stage rollouts, ablation studies, and robotic transfer.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous formulation, clear architectural presentation, and convincing qualitative and quantitative analyses.
Value: ⭐⭐⭐⭐⭐ Provides crucial foundational infrastructure for embodied AI, egocentric video generation, and scalable robot policy simulation.