CausalDrive: Real-time Causal World Models for Autonomous Driving

Conference: ECCV 2026
Paper: ECCV official page
PDF: Full paper
Area: Autonomous Driving
Keywords: World models, driving interaction semantics, causal autoregression, context-forced distillation, policy post-training

TL;DR

CausalDrive generates reactive driving videos from only an initial front-view image, an ego trajectory, and a global interaction prompt; a causal autoregressive teacher and context-forced DMD reduce inference to 1-4 steps, reaching 12.4 FPS on one A100 and an 82.0% yielding rate under the Polite condition on SocioDrive-Bench, without eliminating all false collisions.

Background & Motivation

A driving world model suitable for policy training must answer how nearby vehicles react when the ego vehicle deviates from its recorded trajectory, not merely produce plausible video continuations. Layout-conditioned renderers such as MagicDrive and UniScene receive other vehicles' future positions, making those futures inputs rather than predicted reactions. Log replay has a related problem: the ego vehicle changes course while other vehicles follow their original paths, potentially causing implausible intersections or collisions. Pure action-conditioned predictors avoid future-layout inputs, but may fail to follow ego control precisely and offer limited language control over whether surrounding traffic behaves politely or aggressively.

Real-time interaction makes the problem harder. Bidirectional video diffusion models see an entire temporal sequence and suit offline generation, whereas an interactive simulator must predict successive chunks using only past information, without knowing the user's next action. Simply replacing the teacher with a causal student changes the information available during denoising and leaves accumulated rollout errors unresolved. In particular, when the teacher always receives real history but the deployed student receives its own generated history, good single-step distillation need not produce stable long-horizon behavior.

The paper changes training data, causal architecture, and distillation context together rather than only reducing sampling steps. Core idea: use driving interaction semantics to control reaction distributions, first align the teacher's and student's causal information boundaries, and then require the teacher to supervise the student on its own imperfect history to improve both reactivity and few-step streaming generation.

Method

Overall Architecture

At runtime, the inputs are an initial front-view image, an ego trajectory \(P\), and an interaction prompt \(B\); the output is chunk-wise front-view driving video, without future layouts of other vehicles. Offline, the pipeline constructs interaction-aware data, trains a causal autoregressive teacher, and obtains a few-step student through causal ODE distillation initialization followed by context-forced DMD. The generator can connect to a planner, a video reward module, or human controls, but these applications are not additional visual-generation branches.

The diagram shows training dependencies. Deployment runs the resulting few-step student, without requiring VLM/LLM annotation at every step or online execution of the full teacher.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    DATA["Real driving logs<br/>and safety-critical simulations"] --> SOCIO["Interaction-Semantic Data"]
    SOCIO --> TEACHER["Dual-Condition Causal Teacher"]
    INPUT["Initial front view<br/>ego trajectory and interaction prompt"] --> TEACHER
    TEACHER --> ODE["Causal ODE Distillation"]
    ODE --> DMD["Context-Forced DMD"]
    DMD --> OUTPUT["Few-step streaming student<br/>video and downstream interfaces"]

Key Designs

1. Interaction-Semantic Data: describe who reacts to whom

SocioDrive-Bench contains 20K video clips, with 80% from real nuPlan logs and 20% from CARLA to supplement dangerous and collision events underrepresented in expert driving. Rather than simply annotating vehicles and intersections, it distinguishes ego-initiated pressure, ego-reactive defensive behavior, and complex negotiation. For example, an ego cut-in with a rear vehicle within 15 m becomes an interaction label only when followed by sufficient rear-vehicle deceleration or a sharp reduction in time to collision. These are supervision signals mined through kinematic triggers and temporal associations, not proof of real-world causal identification through randomized interventions.

Annotation has two levels. A VLM processes 4-second left/front/right video clips sampled at 2 Hz and produces structured JSON, separately describing road structure and vulnerable road users so that pedestrian motion is not conflated with vehicle dynamics. An LLM then aggregates consecutive clips into an ego-behavior narrative, a key-interaction summary, and a queryable list of critical events. These respectively smooth temporal discontinuities, supply the interaction prompt, and support dangerous-scenario retrieval for training curricula. Multiple views are available during annotation; this does not mean the deployed world model already generates surround-view video.

2. Dual-Condition Causal Teacher: separate ego geometry from traffic behavior priors

The backbone is initialized from Wan2.1-1.3B. It divides latent video into temporal chunks, allowing bidirectional attention within a chunk for local spatiotemporal consistency while restricting cross-chunk attention to the past and reusing historical KV states. The model can therefore organize the current chunk jointly without accessing future chunks. The paper describes this as \(O(1)\) streaming complexity, but the cache does not specify how an indefinitely growing history is truncated; constant total memory over an infinite horizon cannot be inferred from that claim.

The ego trajectory is converted into a Plücker representation of camera poses, processed by an MLP into geometric modulation parameters, and injected through AdaLN. Interaction text is encoded by a pretrained text encoder and injected through cross-attention to influence other participants' responses. This division matters: the geometry channel makes steering and displacement appear as camera motion, while the semantic channel changes how other vehicles respond under that motion. Polite is a global prior increasing the probability of yielding, not a hard script assigning actions to individual vehicles. AdaLN is also not a strict physical-constraint solver; its control accuracy remains an empirical question.

The teacher uses continuous flow matching and receives only clean, real history when training the current chunk. The cache clearly preserves the linear interpolation path from data to noise:

\[ \psi_t(z_0)=t z_1+(1-t)z_0,\qquad t\in[0,1]. \]

Here, \(z_0\) is a real latent video chunk and \(z_1\) is Gaussian noise. The network learns the path's velocity field, and generation integrates from the noise endpoint back toward data. Causal teacher forcing aligns the information structures of teacher and student; it does not by itself remove the distribution gap between real and generated histories. The fourth design addresses that separate problem.

3. Causal ODE Distillation: initialize the student with information-consistent targets

If a bidirectional teacher can access future chunks during denoising while a causal student cannot, the same current noise under the student's available conditions may correspond to multiple teacher targets obtained using different future information. The paper attributes the resulting blur to a violation of probability-flow ODE injectivity requirements. Its remedy is to sample noise-to-data ODE trajectories from the already trained causal teacher rather than directly distill the original bidirectional model.

Conditioned on real history and controls available through the current chunk, the student regresses the corresponding clean target from intermediate noisy states on those trajectories. The teacher thus does not rely on information unavailable to the student when producing a target. This stage initializes the few-step flow map: first learn sharp generation under appropriate context, then address deviations in student-generated histories. The paper uses approximately 50 steps to illustrate full-teacher latency, but the cache lacks per-experiment solver configurations; this example should not be presented as a fixed 50-step setting for every baseline.

4. Context-Forced DMD: make the teacher face the history the student actually produced

Standard distribution matching distillation can compare different conditional distributions if the teacher's score is estimated using real history while the student rolls forward from its own generated history. CausalDrive first performs an \(N\)-chunk student self-rollout to obtain an imperfect context. The frozen causal teacher must then evaluate the real score using that same context, and the fake-score network is also conditioned on it. The resulting supervision concerns what should come next after this already imperfect history, rather than assuming the past was error-free.

This Self-Corrective Forcing (SCF) does not replace all erroneous past frames with teacher outputs or guarantee recovery from arbitrary severe geometric failures. It exposes training to a history distribution closer to deployment and discourages continued error amplification. The default rollout length is \(N=4\), which must not be confused with 1-4 denoising function evaluations: the former counts historical chunks generated during training, whereas the latter counts inference steps for the current chunk. An adversarial discriminator attached to the fake-score network helps retain high-frequency details such as lane markings, although the paper provides no separate ablation of that discriminator.

A Worked Example

Consider the paper's intersection-creeping example. Keep the initial image and ego trajectory fixed, changing only the interaction prompt. The geometry channel still moves the ego vehicle along the same path; Polite increases the probability that oncoming traffic yields, whereas an aggressive prompt allows it to proceed assertively. The intended control is over different environmental reactions to the same ego action, not over a supplied set of future coordinates for every vehicle. This is a mechanism illustration: the cache does not provide paired, per-instance quantitative differences between those prompts.

After the first chunk, a planner observes the generated image and supplies the next ego-trajectory segment; the student continues generation using historical KV states. For RL, Video2Reward (V2R) converts generated chunks into scalar feedback. The reward expression explicitly available in the cache is:

\[ r_k=R_\psi(x_{k+1})=w_1r_{\mathrm{col}}+w_2r_{\mathrm{lane}}. \]

The terms concern collisions and lanes, but their exact definitions, sign conventions, weights, and V2R supervision are not specified in this cache. The expression explains the feedback interface without establishing a reproducible RL algorithm or a reward calibrated to real accident risk. Human keyboard or steering-wheel inputs can replace planner inputs without changing the generator's conditioning structure.

Loss & Training

The curriculum starts with 1,700 hours of OpenDV front-view video for visual priors, then uses annotated nuPlan and SocioDrive-Bench for interactions, with dangerous Bench2Drive scenarios covering non-expert ego trajectories. Stage 0 uses 8 NVIDIA H20 GPUs, total batch size 4, AdamW, and a constant learning rate of \(1\times10^{-5}\). These settings belong to that stage, not necessarily to every distillation or RL stage.

Training proceeds through base adaptation, a causal flow-matching teacher conditioned on real history, causal ODE trajectory regression, and DMD with student self-rollout and an adversarial texture constraint. Equations (1) and (3)-(6) are visibly damaged in the cached text extraction, so this note does not reconstruct the full flow-matching loss, AdaLN update, or DMD gradient. It retains only the readable interpolation and reward formulas and explains the remaining mechanisms in prose. The paper delegates complete Stage 1/2 hyperparameters to supplementary material absent from the current cache.

Key Experimental Results

Main Results

Table 1 selects the clearly readable FVD and speed columns from the paper's nuPlan reliability comparison. Lower FVD and higher FPS are better. The 12.4 FPS measurement explicitly uses one A100; baseline speeds are approximate, and the cache lacks sufficient detail to establish identical hardware and resolution across methods.

Method	FVD	FPS
Vista	323.37	Approx. 0.3
GEM	291.84	Approx. 0.5
Orbis*	196.21	Approx. 1.2
CausalDrive+	113.6	Approx. 0.5
CausalDrive	121.6	12.4

The cache does not clearly explain the configurations denoted by * and +, so CausalDrive+ is not assigned a specific teacher configuration here. Relative to that row, CausalDrive is faster while FVD changes from 113.6 to 121.6: acceleration is not cost-free on every visual metric.

Reactivity evaluation uses aggressive ego maneuvers. YCR is the frequency with which generated neighboring vehicles correctly decelerate and yield. In the original Table 2, Log-Replay, Vista, and CausalDrive (Polite) achieve YCR values of 0.0%, 12.5%, and 82.0%, with corresponding FCR values of 100.0%, 87.5%, and 18.0%. FCR is the paper's false-collision rate, but complete event-labeling rules are absent from the cache. These rows happen to sum to 100%; that does not establish a general identity of FCR with \(1-\mathrm{YCR}\). The 82.0% result applies to this prompt and evaluation setting, not to a universal probability of traffic yielding.

Ablation Study

Table 2 selects results from the original Table 3, evaluated under the Polite condition on SocioDrive-Bench. \(N\) is the number of student self-rollout chunks.

Configuration	FVD	YCR
Full model	121.6	82.0%
Without interaction prompt	128.4	45.2%
Without causal AR teacher	157.3	38.6%
Without SCF, standard DMD	142.1	72.5%
Only 1 interaction category	133.7	58.3%
Using 2 interaction categories	126.2	73.1%
N=1, no self-rollout	139.8	68.4%
N=2	128.5	77.1%
N=4, default	121.6	82.0%
N=8	120.9	82.3%

Removing the causal teacher reduces YCR by 43.4 percentage points, the largest reduction among the listed single-component removals. Removing the prompt costs 36.8 points; removing SCF costs 9.5 points. Increasing \(N=4\) to \(N=8\) adds only 0.3 points of YCR and improves FVD by 0.7. The paper reports a higher training cost but provides no corresponding GPU-hour measurement. The full model uses all 3 interaction categories, and the category-count ablation also suggests that training-distribution coverage matters.

Key Findings

Table 3 selects policy post-training results on Navsim from the original Table 5. NC is a non-collision metric, TTC is a time-to-collision-related safety metric, Comfort measures comfort, and PDMS is the composite planning score; higher is better throughout. TTC here is an evaluation score, not a duration in seconds.

Policy	NC	TTC	Comfort	PDMS
UniAD	97.8	92.9	100	83.4
DriveDPO	98.5	94.8	99.9	90.0
DiffusionDrive	98.2	94.7	100	88.1
DiffusionDrive-v2	98.4	94.6	100	90.3
RL in CausalDrive	98.7	94.7	100	90.7

Relative to DiffusionDrive, RL in CausalDrive increases PDMS by 2.6 points while tying TTC and Comfort. The paper claims the highest TTC, but DriveDPO's 94.8 exceeds its 94.7, so that claim is not repeated here. Comfort=100 is also shared by several methods. In the original Table 4, ADS is labeled lower-is-better while the larger CausalDrive value is bolded, and its definition is insufficiently documented; this note therefore does not infer superiority on every closed-loop metric from that table.

Highlights & Insights

• Architectural alignment and context alignment are separate problems. First denying future access to both models, then comparing distributions under the same generated history, explains why an attention-mask change or fewer sampling steps alone is insufficient.
• Interaction prompts control a reaction distribution rather than individual future trajectories. This creates a way to obtain different consequences of the same action without giving the renderer the answer, although the causal validity of those reactions still needs independent testing.
• The student's own historical errors can become training conditions. The idea could transfer to long-video prediction or robotic visual simulation, provided the teacher remains reliable under contexts far from real data.

Limitations & Future Work

• The authors explicitly leave multi-camera surround-view generation to future work. Current monocular front-view generation cannot replace complete vehicle-sensor simulation or establish cross-view consistency through occlusions.
• Causality is implemented mainly through temporal restrictions, exclusion of future layouts, and conditional reaction generation. Semantic prompting is not causal identification, and the Polite condition may make the environment overly accommodating, inflating apparent policy safety.
• The evidence supports fewer false collisions, not zero collisions or complete elimination of the ghost effect. Infinite-horizon stability, extreme long-tail cases, reaction calibration across prompts, and confidence intervals remain insufficiently quantified.
• The 12.4 FPS figure measures throughput rather than complete action-to-display latency including planning, cache updates, and decoding. The paper also mentions latency below 100 ms, but the cache lacks the corresponding measurement setup; real-time operation on automotive hardware is not established.
• Damaged equation extraction, missing supplementary material, and inconsistent ADE/FDE versus ADE/Frechet precision/recall terminology limit reproducibility. The table's 0.x precision/recall entries must not be reported as displacement errors in meters.
• V2R training, reward weights, and the complete RL algorithm are insufficiently detailed. Better Navsim scores are not equivalent to on-road vehicle validation; independent reward audits, non-yielding scenarios, and real-interaction comparisons would help identify policies exploiting simulator defects.

Related Work & Insights

• Versus MagicDrive / UniScene: these methods obtain strong rendering control from future layouts, whereas CausalDrive removes that input to predict environmental reactions. They serve different use cases; image quality alone cannot determine which is the better interactive simulator.
• Versus Vista / Orbis: CausalDrive adds interaction semantics to action-conditioned prediction and addresses streaming generation through a causal teacher and few-step distillation. Its reactivity results come from a specific protocol and should not imply that all action-conditioned models lack reactive capabilities.
• Versus DMD / Self Forcing / Causal Forcing: the system builds on distribution matching, generated-history training, and causal-distillation ideas. Its value lies in connecting information and context alignment to driving semantic control, rather than inventing every constituent technique.
• Versus ReSim / log replay: dangerous non-expert trajectories supplement expert-log coverage, while reactive generation attempts to reduce false collisions caused by fixed NPC trajectories. A reusable evaluation strategy is to hold the ego action fixed, vary the environmental reaction distribution, and audit the outputs using independent physical or behavioral criteria.

Rating

• Novelty: 4/5. Combines driving interaction semantics, a causal teacher, and context-forced distillation into an interactive system, with clear precedents for the underlying distillation ideas.
• Experimental Thoroughness: 3/5. Covers speed, reactivity, component ablations, and policy post-training, but lacks full reproduction settings, uncertainty statistics, and on-road evidence.
• Writing Quality: 3/5. The problem formulation is clear, but metric terminology, table directions, and some superiority claims conflict; equation readability is additionally limited by this cache's extraction quality.
• Value: 4/5. Useful for real-time driving generation and stable long-horizon training, but not yet a generally validated safe simulator.