Walk through Paintings : Ego-centric World models from Internet Priors

Conference: ECCV 2026
Paper: ECCV Official Page
Project: egowm.github.io
Area: Image Generation / Embodied World Models
Keywords: World Models, Video Diffusion Models, Action Conditioning, Humanoid Control, Structural Consistency

TL;DR

EgoWM presents an architecture-agnostic conditioning framework that converts off-the-shelf video diffusion models into action-conditioned egocentric world models by injecting compressed, temporally aligned action trajectories directly into the universal denoising timestep pathway, scaling from 3-DoF navigation to 25-DoF humanoid manipulation with high fidelity, extreme OOD generalization into paintings, and introduced the Structural Consistency Score (SCS).

Background & Motivation

Modeling how visual scenes evolve in response to an agent's actions—commonly formalized as visual world modeling—is an essential capability for vision-based navigation, manipulation, and model-predictive trajectory planning. For artificial agents, however, especially embodiments with high degrees of freedom such as humanoids, collecting large-scale action-observation pairs in diverse physical environments is prohibitively expensive. Consequently, existing world models are predominantly confined to narrow simulated environments or customized architectures trained on single-robot domains from scratch, forfeiting the rich physical and visual priors accumulated by Internet-scale video models.

Recent attempts such as Navigation World Models (NWM) train custom autoregressive diffusion models across embodiments for planar navigation, but their bespoke architectures cannot directly leverage Internet-scale video diffusion pre-training and require massive computational resources while still suffering from compounding drift over long horizons. Conversely, fine-tuning approaches like Ctrl-World and GrndCtrl typically rely on global embeddings or ad-hoc cross-attention layers, which often fail to align with the temporal compression inherent in modern Diffusion Transformers (DiTs) and can compromise the pre-trained generative priors.

The core tension lies in bridging data-scarce robotic action spaces with Internet-scale passive video representations without requiring architectural overhauls or sacrificing controllability. This paper observes that every modern video diffusion architecture—whether based on U-Net or DiT—relies on a shared denoising timestep modulation pathway to control the iterative generative process. Core idea: project high-dimensional action trajectories into temporally downsampled latent tokens that match the video VAE's compression factor, and inject them directly into the diffusion timestep modulation pathway alongside an initial proprioceptive state embedding to convert pre-trained video generators into precise, general-purpose world models.

Method

Overall Architecture

EgoWM takes as input an initial observation image \(x_0\) and a multi-step motor action trajectory \(A \in \mathbb{R}^{D \times T}\), where \(D\) represents action dimensionality and \(T\) is the temporal horizon, and generates future rollouts \(\hat{X}_{1:T} \in \mathbb{R}^{T \times H \times W \times 3}\) that strictly follow the action commands. The method preserves the frozen or fine-tuned backbone of the underlying image-to-video diffusion model (such as SVD, Cosmos, or Wan) and interfaces exclusively through a lightweight action projection module and the native timestep modulation layers.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Action Trajectory A (T×D)<br/>3-DoF or 25-DoF"] --> B["Action Projection & Alignment<br/>MLP mapping + 1D conv downsampling to T/k"]
    C["Initial Proprioceptive State Zs<br/>Initial humanoid joint angles"] --> D["Action & Timestep Fusion<br/>Zts + Za (+ Zs)"]
    B --> D
    E["Denoising Timestep ts<br/>Temporally replicated as Zts (T/k×d)"] --> D
    D --> F["Modulation Parameter Generation<br/>Output scale, shift, gate"]
    G["Initial Frame x0 + Latent Diffusion<br/>VAE encoder yields T/k latents"] --> H["Video Diffusion Denoiser Backbone<br/>U-Net / DiT Architecture"]
    F -->|Modulate latent representations| H
    H --> I["Spatio-Temporal VAE Decoder<br/>Decodes T high-resolution future frames"]

Key Designs

1. Action Projection & Latent Temporal Alignment: Resolving Latent Resolution Mismatches Modern video diffusion models, notably scalable DiT backbones, utilize spatio-temporal VAEs with a temporal compression factor \(k > 1\) (e.g., \(k=4\) in Cosmos and Wan) to reduce latent sequence length. Feeding full-rate action vectors directly creates a mismatch with latent frames. EgoWM introduces an action projection module: a sequence of lightweight multi-layer perceptrons (MLPs) maps each \(D\)-dimensional action vector to a \(d\)-dimensional embedding space, followed by two 1D convolutional layers that downsample the sequence along the time dimension to produce \(Z_a \in \mathbb{R}^{d \times (T/k)}\). This guarantees one-to-one temporal correspondence between action embeddings and compressed video latent frames across both 3-DoF navigation \((\Delta x, \Delta y, \Delta \phi)\) and 25-DoF humanoid joint configurations.

2. Action Injection via Timestep Modulation: Preserving Pre-trained Priors While Enforcing Precise Control Instead of adding task-specific cross-attention layers that disrupt pre-trained spatial representations, EgoWM exploits the universal denoising timestep conditioning channel. The global scalar timestep \(t_s\) is embedded and replicated across the compressed temporal horizon to form \(Z_{t_s} \in \mathbb{R}^{d \times (T/k)}\). The aligned action embedding \(Z_a\) is then directly added to \(Z_{t_s}\). For each modulation block \(i\), the affine parameters (scale, shift, gate) are computed via the base model's projection function \(F_i(\cdot)\): $\(P^{\text{scale}}_i, P^{\text{shift}}_i, P^{\text{gate}}_i = F_i(Z_{t_s} + Z_a)\)$ Because modulation acts directly on every latent feature block across temporal coordinates, each latent frame receives frame-specific, action-aligned physical steering while keeping the original generative backbone unaltered.

3. Initial Proprioceptive State Conditioning: Resolving Egocentric Humanoid Occlusions In 25-DoF humanoid control (e.g., EVE 1X), major body parts such as the torso, lower body, and shoulders remain outside the robot's egocentric field of view. Identical delta joint commands can produce vastly different visual trajectories and camera motion depending on the unobserved initial robot posture. To resolve this unobservable ambiguity, EgoWM encodes the initial proprioceptive joint state into an embedding \(Z_s\) via an MLP and incorporates it into the modulation sum: $\(P^{\text{scale}}_i, P^{\text{shift}}_i, P^{\text{gate}}_i = F_i(Z_{t_s} + Z_a + Z_s)\)$ This informs the diffusion model of the robot's physical configuration, enabling accurate distinction between camera-coupled body movements and isolated arm reach actions.

4. Structural Consistency Score (SCS): Disentangling Physical Causality from Visual Appearance Standard video synthesis metrics like LPIPS, DreamSim, and FVD evaluate perceptual similarity and visual fidelity, often rewarding visually sharp frames that nonetheless violate physical actions (such as turning left instead of right). EgoWM introduces the Structural Consistency Score (SCS) to isolate action-following fidelity. SCS first applies dense point tracking (AllTracker) to automatically identify and trim frames where the initial visual scene has completely exited the field of view. For remaining frames, passive scene elements (buildings, furniture, manipulated objects) and visible robot parts are segmented in the initial frame and tracked across both ground-truth and predicted videos using video object segmentation foundation models (such as SAM 3). SCS computes the mean intersection-over-union across \(N\) tracked entities over \(T\) frames: $\(\text{SCS} = \frac{1}{N \cdot T} \sum_{j=1}^{N} \sum_{t=1}^{T} \frac{|\mathcal{M}_{\text{pred}}^{(t,j)} \cap \mathcal{M}_{\text{gt}}^{(t,j)}|}{|\mathcal{M}_{\text{pred}}^{(t,j)} \cup \mathcal{M}_{\text{gt}}^{(t,j)}|}\)$ By focusing strictly on the geometric evolution of passive landmarks, SCS provides an objective measure of whether the predicted visual world evolves consistently with the commanded action.

Loss & Training

EgoWM fine-tunes base video diffusion models using their standard pre-training objectives (noise prediction MSE or flow matching) on noisy latents \(z_{t_s}\): $\(\mathcal{L} = \mathbb{E}_{z, t_s, \epsilon} \left[ \|\epsilon - \epsilon_\theta(z_{t_s}, e_c, t_s)\|^2 \right]\)$ where \(e_c\) incorporates the fused action and state conditioning embeddings. The learning rate for the newly introduced action projection modules is set \(10\times\) higher than that of the base backbone weights to facilitate rapid alignment with the latent space. Fine-tuning runs efficiently on 8 \(\times\) A100 GPUs, using approximately \(8\times\) less training compute than training NWM from scratch.

Key Experimental Results

Main Results

Evaluations span 3-DoF navigation (trained on RECON, SCAND, and TartanDrive; tested on RECON validation) and 25-DoF humanoid loco-manipulation (1X Humanoid Dataset). Backbones include SVD (\(k=1\)), Cosmos-2B (\(k=4\)), and Wan2.1-14B (\(k=4\)).

Table 1: Comparison on RECON Validation Set for 3-DoF Navigation (Table 1 in Paper)

Method	Compression \(k\)	Frame 2 LPIPS↓ / SCS↑	Frame 4 LPIPS↓ / SCS↑	Frame 8 LPIPS↓ / SCS↑	Frame 16 LPIPS↓ / SCS↑
NWM (CVPR 2025)	\(k=1\)	0.26 / 58.4	0.30 / 56.2	0.35 / 46.8	0.45 / 33.4
EgoWM (SVD)	\(k=1\)	0.25 / 62.0	0.27 / 61.7	0.31 / 57.2	0.39 / 55.2
EgoWM (Cosmos-2B)	\(k=4\)	0.20 / 63.1	0.22 / 60.4	0.26 / 56.4	0.33 / 47.5
EgoWM (Wan-14B)	\(k=4\)	0.22 / 62.7	0.26 / 59.0	0.29 / 53.9	0.34 / 49.7

EgoWM variants outperform NWM across all horizons. At Frame 16, EgoWM (SVD) achieves an SCS of 55.2 versus NWM's 33.4 (+65.3% relative improvement), reflecting drastically reduced drift. Cosmos and Wan yield superior perceptual fidelity (LPIPS 0.33-0.34 vs. NWM's 0.45).

Table 2: Results on 1X Humanoid 25-DoF Navigation & Manipulation (Table 2 in Paper)

Task	Variant	Frame 2 LPIPS↓ / SCS↑	Frame 4 LPIPS↓ / SCS↑	Frame 8 LPIPS↓ / SCS↑	Frame 16 LPIPS↓ / SCS↑
Humanoid Navigation	SVD	0.11 / 75.6	0.16 / 66.4	0.26 / 50.3	0.35 / 34.4
Humanoid Navigation	SVD (trained from scratch)	0.17 / 69.7	0.25 / 56.0	0.37 / 40.0	0.47 / 21.6
Humanoid Navigation	Cosmos-2B (\(k=4\))	0.11 / 65.2	0.19 / 54.0	0.27 / 42.3	0.40 / 27.0
Humanoid Navigation	Wan-14B (\(k=4\))	0.09 / 70.2	0.15 / 61.3	0.27 / 45.2	0.39 / 30.2
Humanoid Manipulation	SVD	0.04 / 86.8	0.06 / 81.8	0.10 / 75.0	0.13 / 76.6
Humanoid Manipulation	Cosmos-2B	0.04 / 86.2	0.07 / 82.0	0.10 / 78.2	0.12 / 67.7
Humanoid Manipulation	Wan-14B	0.05 / 81.1	0.07 / 78.5	0.11 / 76.0	0.14 / 65.6

Pre-trained SVD outperforms the scratch-trained SVD baseline on 25-DoF navigation (Frame 16 SCS of 34.4 vs. 21.6; LPIPS 0.35 vs. 0.47), confirming the critical value of Internet-scale video priors in complex physical regimes.

Ablation Study

Table 3: Ablation Analysis at the 16-Frame Horizon (Table 3 in Paper)

Dataset / Task	Variant	LPIPS ↓	DreamSim ↓	SCS ↑	Note
Humanoid Navigation (25-DoF, \(k=4\))	Cosmos-2B (Ours: Timestep Modulation)	0.40	0.19	27.0	Baseline formulation
Humanoid Navigation (25-DoF, \(k=4\))	Cosmos-2B (Global Conditioning)	0.46	0.20	14.9	SCS drops by -44.8%
Humanoid Navigation (25-DoF, \(k=4\))	Cosmos-2B (Chunked Conditioning)	0.42	0.17	23.7	SCS drops by -12.2%
RECON (OOD Zero-Shot)	SVD (Ours: Timestep Modulation)	0.59	0.43	29.7	Trained on SCAND, tested on RECON
RECON (OOD Zero-Shot)	SVD (Ctrl-World: Cross-Attention)	0.64	0.46	22.7	SCS is 23.6% lower; degrades priors

Key Findings

• Timestep Modulation Superiority: Injecting actions into the timestep pathway achieves +23.6% higher OOD SCS than cross-attention conditioning (Ctrl-World), which tends to disrupt pre-trained diffusion priors. In humanoid control, replacing aligned modulation with global conditioning collapses SCS by 44.8% (down to 14.9).
• Temporal Compression Trade-offs: Temporally uncompressed models (\(k=1\)) retain higher structural tracking at long horizons (Frame 16 SCS 55.2 for SVD vs. 47.5 for Cosmos-2B in navigation), whereas DiT models with \(k=4\) produce significantly sharper textures and lower perceptual error (LPIPS 0.33 vs. 0.39).
• Extreme Generalization to Paintings: Pre-trained video priors allow EgoWM to simulate navigation and 25-DoF manipulation inside historical oil paintings (e.g., picking up an apple on a painted table), synthesizing physically consistent camera motion and arm interactions that match the artistic style.
• Downstream Decision Making: Under Cross-Entropy Method (CEM) planning, EgoWM reduces Average Trajectory Error (ATE) in 3-DoF navigation from 3.25 (NWM) to 3.13 (Cosmos). In 25-DoF manipulation, EgoWM-planned actions achieve a 100% success rate on towel placement and 80% on block stacking.

Highlights & Insights

• Universal, Zero-Overhead Adaptation: Adapting video generators via timestep modulation avoids intrusive architectural changes or parameter-heavy cross-attention, allowing instant deployment across U-Net and DiT backbones.
• Metric Innovation with SCS: By evaluating IoU trajectories of tracked passive scene landmarks, SCS provides a rigorous benchmark for physical causality and action alignment without conflating visual crispness with correctness.
• Drastic Training Efficiency: EgoWM trains in 8 A100 days compared to 64 H100s required by scratch-built models like NWM, while achieving up to \(6\times\) faster inference at higher resolution.

Limitations & Future Work

• Temporal Smoothing in Compressed DiTs: 1D temporal downsampling in \(k=4\) latent spaces can attenuate high-frequency micro-adjustments in robotic end-effectors over very short timescales.
• Long-Horizon Extrapolation Drift: Beyond 16 frames (4 seconds), predictions accumulate small geometric inaccuracies and lack global loop-closure mechanisms for multi-minute exploration.
• Complex Multi-Object Contact Physics: Contact-rich interactions involving non-rigid deformations, fluids, or multiple colliding objects can occasionally exhibit mesh penetration or hallucinations.

Related Work & Insights

• vs NWM (Navigation World Models, CVPR 2025): NWM trains a 1B custom autoregressive model from scratch restricted to 3-DoF planar navigation; EgoWM repurposes off-the-shelf foundation models with 8\(\times\) less compute, yielding up to 65% higher SCS and extending to 25-DoF humanoid manipulation.
• vs Ctrl-World (ICLR 2026): Ctrl-World focuses on static-camera tabletop manipulation using cross-attention conditioning; EgoWM addresses first-person dynamic views, handles temporal compression, and maintains superior out-of-distribution robustness via timestep modulation.
• vs GrndCtrl (2025): GrndCtrl relies on global conditioning, which causes an acute loss in fine-grained control; EgoWM enforces strict per-frame temporal alignment, boosting humanoid controllability by over 80% relatively.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Elegant reuse of universal timestep modulation for high-DoF action conditioning across diverse video backbones]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated on 3-DoF and 25-DoF tasks across multiple backbones, introducing SCS and testing OOD paintings and CEM planning]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and thorough empirical validation]
• Value: ⭐⭐⭐⭐⭐ [Establishes a practical, highly scalable blueprint for building general-purpose embodied world models from foundation video models]