DriveVA: Video Action Models are Zero-Shot Drivers¶

Conference: ECCV 2026
arXiv: 2604.04198
Code: None
Area: Autonomous Driving / World Models
Keywords: Video Action Models, End-to-End Autonomous Driving, World Models, Zero-shot Generalization, Flow Matching

TL;DR¶

DriveVA proposes a unified video-action world model that jointly decodes future video prediction and trajectory planning within a single DiT diffusion generation process. Leveraging the spatio-temporal priors of a large-scale video generation model (Wan2.2-TI2V-5B), it achieves strong zero-shot cross-domain generalization, obtaining 90.9 PDMS on NAVSIM, and reducing the average L2 error and collision rate by 78.9%/52.5% and 83.3%/52.4% on nuScenes and Bench2Drive, respectively.

Background & Motivation¶

One of the core challenges in autonomous driving is generalization—models must not only perform well in training scenarios but also remain robust under unseen traffic patterns, road layouts, and sensor configurations. Recently, Vision-Language-Action (VLA) models have alleviated the need for task-specific data to some extent by fine-tuning driving trajectory data on large-scale vision-language models. However, the pre-training of VLAs relies on static image-text pairs, primarily conveying semantic knowledge ("what is what") but lacking the spatio-temporal causal priors ("how the world moves") required for planning. Consequently, true zero-shot cross-dataset generalization remains limited.

On the other hand, large-scale video generation models (e.g., Wan, CogVideoX, HunyuanVideo) have learned realistic motion patterns and physically plausible scene dynamics from massive video corpora, demonstrating strong generalization capabilities to unseen textual and visual contexts. This capability highly aligns with the goal of constructing generalizable driving world models. However, existing world-model-based planning methods suffer from two core bottlenecks: first, the world knowledge learned on a single dataset is difficult to transfer effectively to other datasets, resulting in limited generalization; second, video imagination and trajectory generation are usually modeled separately or only loosely coupled—where visual prediction is processed by one model and trajectory planning by another (or as an auxiliary branch)—leading to an accumulation of inconsistency between visual prediction and planning behavior over time, which causes the executed trajectory to deviate from the future evolution predicted by the world model.

The Key Challenge in this paper is that video generation models possess rich spatio-temporal priors but lack planning capabilities, while driving world models require planning but lack generalization. The Core Idea of DriveVA is to place future video latents and action tokens into a single shared DiT generation process for joint denoising, making the trajectory "action grounded" on the future scene evolution, thereby simultaneously attaining the generalization of video generation and the decision-making capability of planning within a unified generative framework.

Method¶

Overall Architecture¶

The goal of DriveVA is to simultaneously predict the future action sequence (trajectory) and future video frames given historical observation frames, language instructions, and the current ego-status. The entire pipeline consists of three stages: (1) The historical \(m\) observation frames are encoded into a historical latent sequence \(\mathcal{V}_l^{\text{his}}\) by the 3D causal VAE of Wan2.2-TI2V-5B; (2) The language instruction is encoded by a frozen text encoder and injected into the DiT via cross-attention; (3) A unified DiT decoder utilizes the historical latents and ego-status tokens as a fixed conditioning block \(\mathbf{X}_{\text{cond}}^{(l)}\), starting from pure noise to jointly denoise and generate future video latents and action tokens (the target block \(\mathbf{Y}_0^{(l)} = [\mathbf{V}'_{l+1}, \ldots, \mathbf{V}'_{l+n_{\text{pred}}}, \mathbf{A}_{l+1:l+K}]\)). The training and inference interfaces are perfectly consistent: during training, the latents of the full video are split into a conditioning block (noise-free) and a target block (noised for reconstruction), whereas during inference, only the history is encoded and the future is generated from noise. Inference requires only 2 flow matching sampling steps. After executing actions, a sliding window moves the historical buffer to recursively continue video generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Historical Observations + Ego-status + Instructions"] --> B["Video VAE Encoding<br/>Historical Latent + Text Tokens"]
    B --> C["Unified DiT Joint Denoising<br/>Future Video Latent + Action Tokens"]
    C --> D["VAE Decoding → Future Video Frames"]
    C --> E["Action Tokens → Trajectory Sequence"]
    D --> F["Video Continuation: Sliding Window Recursion<br/>Update History for Next Round"]
    E --> F
    F -->|Loop until task ends| A

Key Designs¶

1. Unified Video-Action Joint Generation: Grounding Trajectories on Future Vision

Existing methods (e.g., PWM, Epona, DriveVLA-W0) typically treat video prediction and trajectory generation as two independent or loosely coupled stages: first predicting the future video with one model, and then extracting the trajectory from the predicted video using another model (such as an inverse dynamics model), or adding video prediction as an auxiliary branch alongside the planning model. The fundamental problem with this cascaded design is the lack of deep interaction between visual imagination and planning behavior—the self-attention layers of the DiT cannot see information from "the other side" during denoising, which causes the trajectories to be inconsistent with the generated video scene, accumulating errors during recursive rollouts.

The key design of DriveVA is to concatenate future video latent tokens and action tokens into the same target block, allowing both to interact bi-directionally at every denoising step in the self-attention layers of the DiT. Formally, given the condition \(\mathbf{C}_l = (\mathcal{O}_l, \mathcal{T}, \mathbf{q}_l)\), the joint distribution can be factorized into the product of video continuation and action grounding:

\[\pi_\theta(\mathcal{F}, \mathcal{A} \mid \mathbf{C}_l) = \pi_\theta(\mathcal{F} \mid \mathbf{C}_l) \cdot \pi_\theta(\mathcal{A} \mid \mathbf{C}_l, \mathcal{F})\]

However, DriveVA does not cascade—it denoises the latent tokens of \(\mathcal{F}\) and \(\mathcal{A}\) simultaneously within the same DiT, where self-attention allows visual and action modalities to constantly exchange information during the generation process. This means that the trajectory is not "inferred after watching the video", but is instead produced together during video generation as an intrinsic constraint of the scene evolution.

Ablation studies demonstrate that this design is the core pillar of performance: changing the joint generation to action-only prediction causes the PDMS to plummet from 90.9 to 47.0, showing that future video prediction is not just an auxiliary visualization output but provides a dense temporal grounding for action tokens. Concurrently, changing the attention within the target block from bi-directional to a causal mask (prohibiting action tokens from attending to future video tokens) drops the PDMS from 90.9 to 90.1, showing that even if the conditioning block remains noise-free, cross-modal bi-directional interaction within the target block still yields extra benefit.

2. Video Continuation Strategy: Sliding Window Recursion for Long-term Consistency

Autonomous driving requires continuous planning over long time horizons, but generating an excessively long video all at once is impractical (generating high computational costs and degrading quality over length) and introduces cumulative drift. DriveVA adopts a video continuation strategy: after predicting \(K=8\) steps of actions (covering 4 seconds) and \(N=8\) frames of future video at each step, it executes the action, receives new observations, slides the historical buffer window forward, and uses the latest observational frames as the condition for the next round of generation.

During training, for each complete video segment, the model receives both the historical latent \(\mathcal{V}_l^{\text{his}}\) (conditioning block, unnoised) and future latent \(\mathcal{V}_l^{\text{fut}}\) (target block, noised for reconstruction) after VAE encoding. During inference, it only encodes historical observations \(\mathcal{O}_l\) to get \(\mathcal{V}_l^{\text{his}}\), and the DiT generates the future from pure noise, achieving a perfect alignment of the training and inference interfaces. Sliding window recursion breaks down the difficulty of long-term prediction into a series of short video continuation tasks, ensuring that the model always observes sufficient historical context (\(m=44\) frames, ~2 seconds) to maintain scene consistency.

In ablation studies, removing video continuation (not maintaining video-action coupling during recursive rollout) causes PDMS to decrease from 90.9 to 84.6, validating the crucial role of the continuation strategy in preserving long-term consistency.

3. Large-scale Video Pre-training Prior Transfer: Full Fine-Tuning of Wan2.2-TI2V-5B

The success of DriveVA largely depends on the pre-training priors of the Wan2.2-TI2V-5B large-scale video generation model. Wan has learned rich motion dynamics and physical plausibility priors from massive video data, which are precisely what VLA models (pre-trained on static image-text pairs) lack. DriveVA undergoes full fine-tuning on top of Wan, adapting these general video priors to the task of video continuation and joint action grounding in the driving domain.

The ablation on training strategies clearly demonstrates the value of pre-trained priors and fine-tuning depth: training from scratch yields only 62.9 PDMS, LoRA fine-tuning yields 74.9 PDMS, whereas full fine-tuning reaches 90.9 PDMS. Even when scaling up the model to 14B, using LoRA only achieves 80.6 PDMS, indicating that full fine-tuning of a medium-sized model is more critical for this task than increasing the parameter scale—video priors require deep adjustment of all parameters to transfer effectively to driving planning tasks.

A Complete Example¶

Taking a left-turn scenario in NAVSIM as an example to run through the inference pipeline of DriveVA:

At current time step \(l\), the historical buffer contains 44 historical observation frames (the past 2 seconds @ 22FPS), the language instruction is "turn left", and the ego velocity is \((v_x, v_y)\).
The Wan2.2 VAE encodes the historical frames into conditioning latents \(\mathcal{V}_l^{\text{his}}\), which are concatenated with the ego-status token \(\mathbf{S}_l\) to form the conditioning block \(\mathbf{X}_{\text{cond}}^{(l)}\).
The target block is initialized with pure noise, containing 8 frames of future video latents (corresponding to 4 seconds) and 8 action tokens (each step representing \((x, y, \text{yaw})\)). The text tokens \(\mathbf{T}\) are injected into each layer of the DiT via cross-attention.
The DiT jointly denoises in 2 flow matching sampling steps: noise \(\to\) first-step integration to the intermediate state at \(s=0.5\) \(\to\) second-step integration to the clean target at \(s=1\). A single inference takes approximately 0.1 seconds.
The VAE decoder reconstructs the future video latents into 8 future image frames of size \(832 \times 480\); the action tokens are mapped back to a trajectory via an MLP.
Execute the trajectory, retrieve new observations, slide the historical buffer window forward, and repeat steps 1–5. Over the course of a 15-second scenario, approximately 4 video continuations are generated, with each predicted turning trajectory remaining consistent with the generated left-turn scene evolution.

Loss & Training¶

DriveVA adopts conditional flow matching as the training objective. For each training sample \(l\), starting from the target block \(\mathbf{Y}_0^{(l)}\) (containing future video latents and action tokens), the model samples a flow time \(s \sim \mathcal{U}(0,1)\) and noise \(\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\). It then constructs a linear interpolation \(\mathbf{Y}^{(l,s)} = (1-s)\boldsymbol{\epsilon} + s\mathbf{Y}_0^{(l)}\), with the target velocity being \(\dot{\mathbf{Y}}^{(l,s)} = \mathbf{Y}_0^{(l)} - \boldsymbol{\epsilon}\). The loss function is defined as:

\[\mathcal{L}_{\text{FM}} = \mathbb{E}_{l, s, \mathbf{Y}_0^{(l)}, \boldsymbol{\epsilon}} \left[ \left\| \hat{\mathbf{v}}_\theta^{(l,s)} - \dot{\mathbf{Y}}^{(l,s)} \right\|_2^2 \right]\]

where \(\hat{\mathbf{v}}_\theta^{(l,s)} = f_\theta([\mathbf{X}_{\text{cond}}^{(l)}, \mathbf{Y}^{(l,s)}], s \mid \mathbf{T})\) is the velocity field predicted by the DiT, which acts simultaneously on both the video latents and action tokens, forcing the model to learn their joint distribution during denoising. Training is conducted using AdamW (\(lr=10^{-4}\), weight decay=0.01, bf16 mixed precision) and divided into two stages: first training on a batch size of 80 for 20k steps for fast convergence, and then accumulating gradients to an equivalent batch size of 640 to fine-tune for 10k steps, with a linear warmup for the first 1k steps followed by a constant learning rate. Inference requires only 2 flow matching sampling steps.

Key Experimental Results¶

Main Results (NAVSIM v1)¶

Method	Input	NC↑	DAC↑	TTC↑	Comf.↑	EP↑	PDMS↑
Constant Velocity	-	68.0	57.8	50.0	100	19.4	20.6
Ego Status MLP	-	93.0	77.3	83.6	100	62.8	65.6
UniAD (CVPR'23)	Camera	97.8	91.9	92.9	100	78.8	83.4
DiffusionDrive (CVPR'25)	Camera+Lidar	98.2	96.2	94.7	100	82.2	88.1
LAW (ICLR'25)	Camera	96.4	95.4	88.7	99.9	81.7	84.6
Epona (ICCV'25)	Camera	97.9	95.1	93.8	99.9	80.4	86.2
PWM (NeurIPS'25)	Camera	98.6	95.9	95.4	100	81.8	88.1
WoTE (ICCV'25)	Camera+Lidar	98.5	96.8	94.9	99.9	81.9	88.3
DriveVLA-W0 (ICLR'26)	Camera	98.4	95.3	95.2	100	80.9	87.2
DriveVA	Camera	99.2	97.5	98.7	100	83.5	90.9

Zero-Shot Cross-Domain Generalization (NAVSIM Training, Direct Evaluation)¶

Method	nuScenes Avg L2↓	nuScenes Avg CR↓	Bench2Drive Avg L2↓	Bench2Drive Avg CR↓
DriveVLA-W0	1.43	0.77	3.00	2.52
PWM	3.99	0.36	2.80	3.76
DriveVA	0.84	0.06	1.33	1.79

Ablation Study¶

Configuration	Video Loss	Carla Mix	Video Cont.	NC↑	DAC↑	TTC↑	EP↑	PDMS↑
w/o Video Supervision	✗	✓	✓	95.0	89.0	93.9	59.7	71.4
w/o Carla Mix	✓	✗	✓	99.0	97.3	98.4	83.2	90.5
w/o Video Continuation	✓	✓	✗	94.9	95.6	94.2	76.9	84.6
Full Model	✓	✓	✓	99.2	97.5	98.7	83.5	90.9

Key Findings¶

Video supervision is the primary driver of performance gains: Removing the Video Loss causes the PDMS to plummet from 90.9 to 71.4 (-19.5), far exceeding any other ablation. Dense, video-level temporal supervision provides the key world dynamics grounding for action tokens, rather than being merely an auxiliary loss.
Only 2 sampling steps are required to reach optimality: 1 step yields only 13.2 PDMS, 2 steps reach 90.9, and 3 steps provide no extra gain. The video pre-training priors place the starting point of denoising close to the data distribution, which dramatically reduces inference overhead and makes generative planning practically feasible in autonomous driving.
An optimal value exists for the number of predicted video frames: predicting 8 frames of future video (covering 4 seconds) yields the highest PDMS (90.9), while 4 frames drops to 82.1 (insufficient coverage), and 12 frames drops to 86.7 (increased drift).
Full fine-tuning is far superior to LoRA: training from scratch yields 62.9, LoRA yields 74.9, and full fine-tuning achieves 90.9 PDMS. Even with a 14B model, using LoRA only yields 80.6, indicating that full fine-tuning of a medium-sized model is more important than expanding the parameter scale for this task.
Zero-shot capability stems from maintaining video-trajectory consistency: Qualitative analysis (Figure 3) shows that PWM often suffers from inconsistency between video imagination (turning left) and trajectory (going straight) in zero-shot scenarios, whereas DriveVA consistently keeps them aligned, which is the direct source of its cross-domain generalization capability.

Highlights & Insights¶

The experimental finding that "video prediction is not an auxiliary output, but a grounding for planning" is highly valuable: Many prior works treat video prediction merely as a nice-to-have visualization aid. DriveVA powerfully demonstrates through ablation studies (removing video loss causes a 19.5 drop in PDMS) that video prediction itself is a key source of planning capability—it provides dense temporal grounding for actions, rather than being an optional add-on.
The extremely low sampling steps (2 steps) of Flow Matching make generative planning truly practical: Diffusion and flow matching methods are often criticized for their high inference overhead, making them unsuitable for real-time systems. However, DriveVA discovers that the video pre-training priors allow the denoising start point to be close to the data distribution, achieving convergence in just 2 steps, which paves the way for the real-time deployment of generative world models in autonomous driving.
The unified joint denoising design can be transferred to other multi-modal alignment tasks: The idea of placing visual prediction and action generation in the same DiT for bi-directional interaction can be generalized to scenarios such as robotic manipulation (joint video-action prediction) and embodied navigation (joint vision-control generation), addressing the inconsistency between visual imagination and action planning.
Using DPVO as an external validation tool to evaluate video-trajectory consistency is highly inspiring: Independently running visual odometry to reconstruct trajectories from generated videos and comparing them with the model's predicted trajectories offers a ground-truth-free consistency evaluation method. This can be extended to other world model evaluations, providing a quantifiable answer to whether the "generated video is trustworthy."

Limitations & Future Work¶

Errors at the causal or intentional level cannot be resolved by video-trajectory consistency: Failure cases presented in the paper (e.g., stopping conservatively when bypassing is required, predicting a stop when proceeding through an intersection is appropriate) show that when the model's intent judgment for the future scene is itself wrong, both the video and the trajectory will be consistently wrong—i.e., aligned in direction, but incorrect in decision. This indicates that future work should not only focus on vision-action alignment but also improve the model's causal scene understanding and multi-modal future reasoning capabilities.
Using only a front-facing monocular camera: The current version relies solely on front-view inputs and does not utilize additional information such as multi-view systems, LiDAR, or HD maps. Incorporating multi-modal inputs could further improve safety-critical metrics (such as NC, TTC), especially under occlusion or extreme lighting conditions.
Limited closed-loop evaluation: The closed-loop testing on Bench2Drive Dev10 only includes a small number of routes, lacking a complete Bench2Drive closed-loop assessment. Furthermore, NAVSIM itself is an open-loop PDM evaluation, meaning that true closed-loop performance requires more comprehensive validation.
The reality gap from real-world to simulation remains significant: The collision rate of 1.79% on Bench2Drive is much higher than the 0.06% on nuScenes, indicating that substantial differences in appearance, dynamics, and agent interaction behaviors still exist between real logs and the CARLA simulator, leaving room for improvement in zero-shot cross-domain transfers.

vs. PWM / Epona / DriveVLA-W0 (World-Model-Based Planning Methods): These methods treat video prediction and trajectory generation as independent or loosely coupled modules, where video-action inconsistency is a common challenge. The core innovation of DriveVA lies in unifying both into a shared generation process for joint denoising, fundamentally eliminating inconsistency. Experiments show that this design yields significant zero-shot generalization gains.
vs. VLA Methods (DriveMoE / OpenDriveVLA): VLA models rely on the static semantic priors of vision-language models, improving robustness by constructing corner-case data or through expert division of labor. DriveVA takes a fundamentally different path—inheriting spatio-temporal dynamic priors from video generation models, demonstrating a qualitative leap in zero-shot generalization (surpassing fully fine-tuned SOTA methods on nuScenes without any fine-tuning).
vs. LAW / AdaWM / World4Drive (Implicit World Models): These methods learn dynamics in a compact latent space for planning without generating explicit videos. DriveVA opts for explicit future video generation, which gains the advantage of allowing external tools like DPVO to independently verify vision-trajectory consistency, while video-level dense supervision offers richer learning signals for planning.
vs. VaViM / VaVAM (Video Action Models): VaViM similarly explores using video diffusion models for driving. However, DriveVA goes a step further by jointly generating videos and actions inside a single shared DiT (rather than separately or aligned in post-processing), and thoroughly investigates how data scale and diversity affect generalization, providing clear ablation evidence.

Rating¶

Novelty: ⭐⭐⭐⭐ The core concept of introducing joint video-action generation into autonomous driving world models is novel and insightful; however, the flow matching and DiT architectures themselves are existing technologies, making the core contribution lie in "combination and adaptation" rather than fundamental methodology innovation.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Cross-domain zero-shot evaluation on three benchmarks (NAVSIM, nuScenes, and Bench2Drive), multi-dimensional ablations (video supervision/continuation/Carla mix/sampling steps/predicted frame count/training strategy/model scale/mask strategy), and DPVO external consistency verification make the experimental design comprehensive and convincing.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation of the problem, detailed description of the method, rigorous experimental logic, and rich charts. The contradiction-driven narrative gives readers a clear understanding of "why it is done this way."
Value: ⭐⭐⭐⭐⭐ The core finding that video prediction is a planning grounding rather than just an auxiliary output, the practical applicability of generative planning achieved via ultra-low sampling steps, and the robust zero-shot generalization capability provide significant promotional value to the field of autonomous driving world models.