Learning Transferable Dynamics Priors from Action to World Modeling

Conference: ECCV 2026
arXiv: 2606.29501
Code: https://github.com/LogosRoboticsGroup/A2World
Area: Video Generation
Keywords: Diffusion World Models, Robotic Manipulation, Action-to-Video Generation, Transferable Dynamics Priors, Multi-view Generation

TL;DR

This work proposes A2World, which pre-trains an action-conditioned multi-view diffusion world model on large-scale real-world robot manipulation data (2.156 million trajectories, 20+ robotic arm morphologies). The learned action-to-dynamics prior can be adapted either into a long-horizon simulator (A2World-sim) or an instruction-conditioned policy (A2World-policy), achieving state-of-the-art performance on the LIBERO benchmark and real robots.

Background & Motivation

Recently, the field of robot learning has increasingly adopted video generation models as basic backbones. These efforts generally evolve along two trajectories: first, adapting video models into Vision-Language-Action (VLA) policies that drive action generation through video prediction (e.g., Cosmos Policy, DreamZero, LingBot-VA); second, developing action-conditioned world models for data augmentation and policy evaluation (e.g., Ctrl-World, DreamDojo, World4RL), or even combining them with reward models for policy post-training. Despite significant progress, these methods have not fully tapped into the potential of robot data pre-training as a source of transferable dynamics priors. Most works directly fine-tune from general video generation checkpoints (e.g., Cosmos-Predict2), lacking a pre-training phase with actions as supervision signals on large-scale real-robot annotated data. Even works pre-trained on large-scale datasets are often optimized for a single downstream objective (either simulation or policy), rather than being explicitly designed as reusable dual-purpose priors.

The Key Challenge of this gap lies in: actions in manipulation tasks inherently provide natural causal supervision signals—low-level physical interaction rules like contact, grasping, pushing, and releasing are shared across different objects, scenes, and viewpoints. However, existing methods either use text conditioning to approximate dynamics (which is less precise) or generate indirect pseudo-labels with latent action models (facing domain shifts). The Key Insight of this paper is: given the recent emergence of large-scale high-quality open-source robotic manipulation datasets such as AgiBot, DROID, InternData, and RoboCoin, the conditions are ripe for utilizing real-world action annotations within a unified framework to learn "how actions drive the evolution of visual scenes".

Core Idea: This paper proposes A2World, which pre-trains an action-conditioned diffusion world model on large-scale, multi-morphology (20+ robotic arm configurations), multi-task, and multi-view real robotic manipulation trajectories. It explicitly designs the learned action-to-visual dynamics prior as a dual-purpose paradigm: adapting both as a task-specific long-horizon simulator (A2World-sim) for policy evaluation and counterfactual analysis, and as an instruction-driven video-action joint prediction policy (A2World-policy) directly used for robotic manipulation.

Method

Overall Architecture

A2World is the core pre-trained model of this paper, positioned as an action-conditioned multi-view latent diffusion world model. The inputs are a single-frame initial observation \(o_t\) and a future action chunk \(a_{t+1:t+k}\) of length \(k=20\), and the outputs are the corresponding future \(k\) frames of multi-view video \(o_{t+1:t+k}\). The model is based on the DiT (Diffusion Transformer) architecture and operates on a continuous latent space encoded by the WAN2.1 tokenizer, initializing the backbone network from Cosmos-Predict2-2B-Video2World. After pre-training, the same set of weights is adapted downstream in two ways: first, it is fine-tuned as A2World-sim by incorporating a history-aware mechanism, supporting long-horizon autoregressive rollouts to serve as an alternative simulation environment for real-robot training and evaluation; second, it is fine-tuned as A2World-policy by integrating text instruction conditioning and an MoE-like joint video-action module, outputting action sequences to directly drive the robotic arm.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Initial Frame + Action Chunk"] --> B["Action Conditioning Injection<br/>MLP Encoding + Timestep Modulation"]
    B --> C["Multi-view Joint Generation<br/>View Embedding + Cross-view Attention"]
    C --> D["Pre-trained Prior"]
    D -->|Simulation Adaptation| E["A2World-sim<br/>Pose-guided History Sampling<br/>+ Dual-path Injection + Self-forcing"]
    D -->|Policy Adaptation| F["A2World-policy<br/>MoE Video-Action Joint<br/>+ T5 Instruction Conditioning + Modality CFG"]

Key Designs

1. Action Conditioning Injection: Diffusion Prior Learning with Action as the Sole Conditioning Signal

How to ensure the diffusion model learns "action-driven dynamics" rather than "mere continuation of scene appearance"? The most fundamental design choice of A2World is: setting all text conditioning features to zero \(\mathbf{c}=\mathbf{0}\) in the pre-training stage, while retaining only action conditioning. Specifically, the action chunk \(a\) is encoded via an MLP into an action embedding \(e = \text{MLP}(a)\), which is then added to the diffusion timestep embedding. This allows the scale/shift/gate modulation parameters generated by the Adaptive Layer Normalization (AdaLN) of each DiT block to simultaneously incorporate both timestep and action information:

\[\tilde{\tau}(\sigma) = \tau(\sigma) + e\]

This design cuts off the model's shortcut of predicting the future via text cues—it cannot predict "the object is red so where should it be moved", but must instead learn "how this 7-dimensional end-effector pose increment will cause changes in specific pixels of the scene", truly coupling visual dynamics with physical actions. Experiments also corroborate this finding: action-to-video pre-training significantly outperforms text-conditioned pre-training in terms of downstream policy performance (LIBERO average 98.6% vs 97.4%), as the uncertainty in text conditioning corresponding to multiple valid action sequences is completely eliminated.

2. Multi-view Joint Generation: Temporal Concatenation + View Embedding + Cross-view Attention

Real-world robotic operations typically involve multiple cameras. A2World temporal-concatenates frames from all \(V\) viewpoints into \(\mathbf{z}_{\text{mv}} \in \mathbb{R}^{B \times C \times (V \cdot T) \times H \times W}\), which effectively acts as a long video sequence with consecutive viewpoints. To enable DiT to distinguish the identity of different viewpoints, a learnable view embedding \(\epsilon_{\text{view}}(v) \in \mathbb{R}^{d_e}\) is designed for each view and concatenated with the latent features in the channel dimension. Crucially, a cross-view attention module is inserted into each DiT block: tokens of a particular view can attend to the tokens of all other views, thereby capturing the spatial correspondences across views. This joint generation design avoids spatial inconsistency overhead caused by independent frame generation of each viewpoint, while reserving viewpoint-specific details (e.g., lighting variations, distinct occlusion patterns), providing physically consistent multi-view information for subsequent policy evaluation and manipulation.

3. Pose-guided History Sampling and Dual-path History Injection (A2World-sim)

Adapting the short-horizon A2World into a long-horizon simulator presents two challenges. The first is the selection strategy for historical memory: keeping only the most recent frames in a sliding window loses early critical interactions (such as the object state before grasping), while keeping all frames exceeds the token budget. The pose-guided history sampling of A2World-sim calculates the weighted arc length of the robotic arm movement based on relative actions, and uniformly samples \(m\) frames in the arc-length space to cover the key states of the movement with a fixed budget—including the moments preceding contact and turning points. The second is the injection method of historical information: a dual-path strategy is adopted. On one hand, historical tokens replace the empty cross-attention conditioning from the pre-training stage to let the current frame tokens retrieve historical context via cross-attention. On the other hand, historical tokens are concatenated as key/value memory in the self-attention of the current latent token to provide global state memory. In addition, A2World-sim employs a Self-forcing training strategy: during training, self-generated frames, instead of ground-truth frames, are periodically used as conditions, exposing the model to its own rollout errors and teaching it to recover from them. This approach does not require an independent teacher model, since given the actions and the initial frame, the future trajectory is largely governed by underlying physics. Though self-generated frames contain errors, they still convey the correct trend of dynamics, allowing the trained model to show significantly better stability in long-horizon rollouts.

4. MoE-like Video-Action Joint Module (A2World-policy)

The core problem of transferring world model priors to action policies is: how can video dynamics priors be translated into action generation capabilities? The design of A2World-policy features an MoE-like shared attention + modality-split branch structure—video tokens and action tokens share the same self-attention module (allowing action generation to "borrow" the pre-trained video dynamics priors), while having independent AdaLN and MLP branches (preserving modality-specific denoising capabilities). This avoids the massive data overhead of training action branches from scratch and prevents catastrophic interference of action prediction on video dynamics. The instruction text is encoded by T5 and injected into each DiT block as a cross-attention condition. During inference, modality-wise classifier-free guidance (modality-wise CFG) is adopted, where video and action use separate guidance scales \(s_v, s_a\) to flexibly balance visual quality and action accuracy. The joint denoising objective is:

\[\mathcal{L}_{\text{A2World-policy}} = \mathbb{E}\left[ w(\sigma_v) \|\hat{\mathbf{z}}^v - \mathbf{z}^v \|_2^2 + \lambda_a w(\sigma_a) \|\hat{\mathbf{z}}^a - \mathbf{z}^a \|_2^2 \right]\]

where video and action employ coupled shared noise levels \(\sigma_v = m_v \sigma_{\text{base}}, \sigma_a = m_a \sigma_{\text{base}}\) (\(m_v=\sqrt{6}, m_a=0.5\)) to maintain temporal alignment of video and actions during joint training. The experiments in the paper also reveal an interesting positive coupling phenomenon: the quality of video prediction is highly positively correlated with the quality of action generation during training; joint training achieves a much higher upper bound than freezing the video branch.

Loss & Training

A2World pre-training adopts the EDM denoising score matching loss, while A2World-policy employs the weighted joint denoising loss (video and action loss weight \(\lambda_a=1\), with the final backpropagation gradient scaled by 10x). All steps utilize the fused Adam optimizer with a learning rate of 1e-4 and a weight decay of 0.1. A2World pre-training is conducted on 64 H200 GPUs with a batch size of 12 per GPU and gradient accumulation of 4, training for 2 epochs. A2World-policy fine-tuning uses 32 H200 GPUs with a global batch size of 256 for 20k steps. The diffusion process utilizes a rectified flow with \(\sigma_{\min}=4.0, \sigma_{\max}=80.0, \rho=7.0\), and 35 sampling steps.

Key Experimental Results

Main Results

LIBERO Policy Success Rate Evaluation: A2World-policy achieves an average success rate of 98.6% under the standard four-suite protocol, which is the highest among all compared methods, particularly showing outstanding performance in long-horizon (Long) tasks (98.2%).

Method	Spatial	Object	Goal	Long	Average
Diffusion Policy	78.3	92.5	68.3	50.5	72.4
OpenVLA-OFT	97.6	98.4	97.9	94.5	97.1
Cosmos Policy	98.1	100.0	98.2	97.6	98.5
A2World-policy	98.2	99.2	98.6	98.2	98.6

Simulator Rollout Quality Evaluation (LIBERO Dataset): A2World-sim comprehensively outperforms baselines such as Cosmos-Predict2, Ctrl-World, and Prophet across visual quality and action fidelity metrics.

Method	PSNR↑	SSIM↑	tSSIM↑	EPE↓	cos↑
Cosmos-Predict2	25.36	.8792	.7631	.4009	.5755
Ctrl-World	23.60	.8632	.7445	.6827	.3730
Prophet	26.12	.8887	.7789	.3667	.5932
A2World-sim	26.64	.8957	.7862	.3498	.6045

Ablation Study

Ablation of History Sampling Strategies (LIBERO Dataset): Pose-guided sampling significantly outperforms both the no-history strategy and sliding window baselines across all metrics, validating the importance of motion keyframe selection for long-horizon rollout stability.

Configuration	PSNR↑	SSIM↑	tSSIM↑	EPE↓	cos↑
No History	25.41	.8806	.7663	.3969	.5778
Sliding Window	25.63	.8840	.7699	.3900	.5853
Pose-guided (Ours)	26.64	.8957	.7862	.3498	.6045

Ablation of Pre-training Variants (LIBERO Policy): Action-to-video pre-training (A-pre) achieves 98.6%, vastly outperforming text-conditioned pre-training (T-pre, 97.4%) and Cosmos initialization (C-init, 97.0%). It is close to policy-oriented pre-training (P-pre, 98.8%), but uniquely retains simulator reuse capability.

Configuration	Average Success Rate
Text-conditioned Cosmos Initialization (C-init)	97.0
Text-conditioned A2World Pre-training (T-pre)	97.4
Action-to-video A2World Pre-training (A-pre)	98.6
Policy-oriented Pre-training (P-pre)	98.8

Key Findings

• Action-to-video pre-training significantly outperforms text-conditioned pre-training in downstream policies because visual transitions are almost completely determined by actions, eliminating the uncertainty of text instructions corresponding to multiple valid action sequences.
• There is a positive coupling between video modeling and action learning: during training, higher video prediction quality correlates with higher action generation quality. Joint training reaches a higher upper bound than freezing the video branch, showing that the shared self-attention module effectively transfers dynamics priors.
• In OOD (out-of-distribution) scenarios, A-pre substantially outperforms C-init (LIBERO-Plus Spatial average of 88.5% vs 80.2%), demonstrating that action-to-video pre-training captures physical dynamic interaction priors beyond mere visual appearances.
• The evaluation of A2World-sim as a real-world simulator shows high alignment with the real world (Spearman \(\rho=0.916\), Pearson \(r=0.965\), \(R^2=0.930\)), validating the feasibility of utilizing world model rollouts as a substitute for real-robot rollouts in policy evaluation.

Highlights & Insights

• The pre-training design with action as the sole conditioning signal is highly ingenious: setting all text conditioning to zero forces the model to learn the causal mapping from actions to visual dynamics in the latent space, avoiding the shortcut of "guessing the future from visual frames" common in general video models. This is the core design philosophy of this paper and the root of various benefits.
• The pose-guided history sampling algorithm is simple and efficient: by computing the motion arc length and uniformly sampling within the arc-length space, it maximizes the information density of history frames within a fixed budget. It is more robust than simple sliding windows without adding extra computational overhead (no increase in token counts).
• The MoE-like shared attention + modality-split branch structure is an elegant transfer design: sharing the self-attention allows action generation to "borrow" the dynamics prior learned from video pre-training, while the modality-specific independent branches preserve their own denoising traits, preventing cross-modality interference.
• The Self-forcing training strategy incorporates exposure to long-horizon rollout errors into the training process without needing an independent teacher model, fitting the action-conditioned prior perfectly.
• The "pre-train once, adapt twice" paradigm is highly practical: the exact same A-pre checkpoint can be utilized for either simulator or policy initialization. Its performance closely aligns with policy-oriented pre-training (P-pre) specifically optimized for policy (98.6% vs 98.8%), yet it adds a powerful simulator adaptation capability.

Limitations & Future Work

• Pre-training relies solely on tabletop robotic manipulation data (e.g., AgiBot, DROID, InternData), without covering broader embodiments like mobile manipulation or humanoid robots. Extending the dynamics prior to richer locomotion and interaction modes is a natural future extension.
• The models are parameter-heavy (A2World 2.5B, A2World-policy 3.0B), leading to high inference overhead, which is still a distance away from real-time deployment on consumer-grade robot hardware. Exploration of knowledge distillation or sparse MoE presents potential paths forward.
• The OOD generalization evaluation is conducted on LIBERO-Plus Spatial, and the real-world simulator consistency evaluation only covers 8 policy-task combinations. Validation under broader scenarios and degraded conditions (sensor noise, sudden lighting changes) remains to be performed.
• A2World-policy utilizes a diffusion model to generate actions, which is slow in inference. Whether this can be substituted with consistency models or multi-step distillation schemes is a research direction worth exploring.

Related Work & Insights

• vs Cosmos-Predict2: A general video generation model with text conditioning but no action signals; A2World is pre-trained with action supervision on large-scale robotic data, leading significantly in action fidelity metrics (EPE 0.3498 vs 0.4009).
• vs Ctrl-World / Prophet: Similar action-conditioned diffusion world models. However, Ctrl-World's pre-training data is limited to DROID, and Prophet is single-view; A2World obtains superior generalization through joint training on multiple datasets, multi-morphologies, and multi-views.
• vs DreamDojo: Also acts as a simulator on LIBERO, but lacks large-scale action-to-video pre-training, making OOD scenarios more prone to drifting toward the visual appearance of the training domain.
• vs Cosmos Policy / LingBot-VA: Video-to-action policies, but their pre-training does not exploit action conditions. A2World-policy shows distinct superiority on long-horizon tasks (LIBERO Long 98.2% vs Cosmos Policy 97.6%), and can also function as a simulator using the same prior.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [First to systematically present the paradigm of pre-training action-to-video diffusion world models as transferable dynamics priors, with a clear dual-purpose adaptation logic and a simple, elegant design]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Covers LIBERO/LIBERO-Plus/RoboNet simulators and 5 real-robot tasks, including simulator consistency evaluations, OOD transfer, comprehensive ablations, and comparisons against multiple baselines]
• Writing Quality: ⭐⭐⭐⭐⭐ [The narrative transition from pre-training to the dual adaptations flows naturally, with clearly articulated research motivations and well-structured experimental presentations and tables]
• Value: ⭐⭐⭐⭐⭐ [Provides a new "pre-train once, adapt twice" paradigm for the robotics field, which practically aids in reducing real-robot training costs and enhancing policy generalization]

Related Papers

• [ECCV 2026] CustomX: Unified Character, Action, and Scene Customization in Video World Models
• [ECCV 2026] MemLearner: Learning to Query Context Memory for Video World Models
• [ICML 2026] OLAF-World: Orienting Latent Actions for Video World Modeling
• [CVPR 2026] SeeU: Seeing the Unseen World via 4D Dynamics-aware Generation
• [CVPR 2026] Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamics