Thinking Ahead: Foresight Intelligence in MLLMs and World Model¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://huggingface.co/datasets/Gong-Grant/FSU-QA
Area: Multimodal VLM / LLM Reasoning
Keywords: Foresight Intelligence, Multimodal LLMs, World Models, Counterfactual Reasoning, Autonomous Driving

TL;DR¶

The authors introduce FSU-QA and its benchmark FSU-Bench to evaluate foresight intelligence in autonomous driving across 9 hierarchical cognitive tasks over a 12-second horizon, establishing a VLM proxy probing paradigm that validates whether world-model-generated futures convey physically and semantically consistent outcomes.

Background & Motivation¶

Vision-Language Models (VLMs) have demonstrated substantial success in 2D/3D visual perception, spatial grounding, and embodied decision-making. However, existing benchmarks in autonomous driving—such as NuScenes-QA, DriveLM, and DriveGPT4—predominantly emphasize present-scene understanding, static property identification, or reactive path planning within brief temporal windows. These setups only test whether a model knows what is visible right now or what immediate action the ego vehicle should execute in the next second. Real-world autonomous navigation demands Foresight Intelligence: the cognitive capability to model, extrapolate, and anticipate long-horizon, uncertain future outcomes and simulate hypothetical "what-if" counterfactual scenarios conditioned on past observations.

Current benchmarks suffer from two major deficiencies in addressing this capability. First, conventional driving VQA datasets focus on immediate short-horizon actions and are heavily skewed by the "safety bias" inherent in benign naturalistic driving datasets (e.g., nuScenes), where near-collision or safety-critical conflicts rarely occur; even when counterfactual queries are introduced (such as in OmniDrive), evaluation is typically grounded in geometric waypoint displacement errors (L2 loss) rather than natural language semantic causal deductions. Second, while contemporary generative World Models (WMs) synthesize visually convincing future video rollouts, community consensus remains lacking on whether these synthetic futures reflect true physical dynamics and semantic fidelity, since standard generative metrics like FVD or FID only capture low-level textural coherence rather than functional downstream utility.

To bridge this gap, this paper unifies vision-language foresight reasoning with world model validation. Core idea: build FSU-QA, a multi-agent counterfactual QA dataset spanning a 12-second future window across 9 hierarchical cognitive tasks, and deploy VLMs as discriminative proxy probes alongside shuffled control experiments to evaluate both MLLM foresight capabilities and the semantic coherence of world models.

Method¶

Overall Architecture¶

FSU-QA is built upon 850 real-world driving sequences from nuScenes (each approximately 15 seconds long). For every sequence, the front-view monocular camera recording is segmented into a 3-second historical observation window ($t \in [-T_h, 0]$) and a 12-second future evolution window ($t \in [1, T_f]$). The overarching methodology is anchored by three cognitive pillars: Situational Modeling, Causal and Dynamic Simulation, and Goal-Oriented Evaluation. First, high-definition (HD) maps, bounding-box trajectories, and ego kinematics are parsed to construct spatio-temporal representations. Second, a question checklist spanning 9 tasks across three cognitive complexity tiers (low-level spatio-temporal dynamics, mid-level VRU risk assessment, and high-level counterfactual prediction) is populated via template-based synthesis, with counterfactual ground truths derived via closed-world geometric intersection checks and verified by expert human-in-the-loop auditing. Third, the benchmark supports a dual-track evaluation: VLM-oriented evaluation measures zero-shot foresight inference directly from history, while WM-oriented evaluation injects world-model-generated future video rollouts and trajectories into the VLMs to quantify the semantic fidelity of generative world models via downstream QA performance gains.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: 3s front-view historical video + ego trajectory"] --> B["Hierarchical Cognitive Task Suite<br/>Low-level Spatio-temporal / Mid-level VRU Risk / High-level Counterfactual"]
    B --> C["Closed-world Counterfactual Annotation & Human-in-the-Loop Review<br/>4D intersection checks + 3-round expert audit"]
    C --> D["World Model Semantic Probing via Proxy Evaluator<br/>Future rollout injection + shuffled control validation"]
    D --> E["Output: Motion trend forecast / Collision consequence prediction / WM semantic utility score"]

Key Designs¶

1. Hierarchical Cognitive Task Suite: structuring multi-level foresight capabilities across a 12-second horizon

To systematically probe predictive capacity rather than surface-level patterns, the benchmark structures 9 tasks into three distinct cognitive tiers: - Low-level Spatio-temporal Dynamic Reasoning (72.94% of dataset): Evaluates long-horizon multi-agent dynamic extrapolation across five sub-tasks: Speed Change, Turn Change, Lane Change, Relative Distance (discretized into spatial bins), and Relative Position. By dividing the 12-second future into successive 3-second intervals, models are compelled to predict sequential ego-vehicle motion and surrounding agent dynamics without access to future visual frames; - Mid-level VRU-centric Risk Assessment (7.76% of dataset): Focuses on vulnerable road users (VRUs) and safety-critical hazard detection via three tasks: Pedestrian Intent (predicting crossing, walking, or standing over the next 0–3s), Ego-VRU Relative Position (relative bearing and placement at the 3-second mark), and Risk Area (categorizing environmental risk levels); - High-level Causal Reasoning (Counterfactual Prediction, 19.29% of dataset): Overcomes dataset safety bias by querying non-executed hypothetical actions (e.g., "If the ego vehicle were to turn left with a constant radius over the next 3 seconds, what would be the most likely outcome?"), evaluating whether models can simulate collisions, boundary violations, or safe maneuvers under hypothetical maneuvers.

2. Closed-world Counterfactual Annotation and Human-in-the-Loop Review: simulator-free rigorous ground truth construction

Synthesizing credible ground truth for counterfactual actions without a physics simulator is an open challenge. The authors solve this by leveraging the closed-world geometric constraints of nuScenes: - Kinematic and Geometric Intersection Checking: Utilizing precise 3D object bounding-box annotations, temporal agent trajectories, and HD-map lane/boundary polylines, hypothetical maneuvers (e.g., prescribed acceleration or fixed-radius turns) are mapped into explicit 4D spatio-temporal swept volumes. Spatial intersections between the projected ego volume, surrounding dynamic agents, and static road boundaries rigorously determine whether a collision or road departure occurs; - Multi-round Expert Quality Control: To guard against edge-case geometric failures, three domain experts independently audited 45 sampled scenes (~30% of FSU-Bench) across three rounds, categorizing issues into label logic errors, corner-case misclassifications, and question ambiguities. The review resulted in revising or pruning 654 QA pairs, ultimately achieving an 89% human-expert agreement rate (Cohen's $\kappa = 0.78$) and reducing human-flagged error rates below 5%.

3. World Model Semantic Probing via Proxy Evaluator: quantifying generative fidelity through downstream QA utility

Standard generative evaluation metrics (FVD, FID) fail to determine whether world models capture physical causality and scene semantics. The authors introduce a proxy evaluation protocol that employs VLMs as semantic probes: - Formulation: The standard baseline queries the VLM with historical observations $\hat{a} = \text{VLM}(q, V_{-T_h:0}, \text{Traj}_{-T_h:0})$, whereas the WM-augmented setting incorporates future predictions synthesized by a world model $\hat{V}_{1:T_f}$ and $\hat{\text{Traj}}_{1:T_f}$: $$\hat{a} = \text{VLM}\left(q, V_{-T_h:0}, \text{Traj}_{-T_h:0}, \hat{V}_{1:T_f}, \hat{\text{Traj}}_{1:T_f}\right)$$ Improvements in downstream QA accuracy directly reflect the semantic utility and physical fidelity of the predicted futures; - Shuffled Control Validation: To ensure that downstream gains originate from genuine predictive content rather than generic visual token padding, a negative control is implemented by pairing queries with predictions generated from randomly selected mismatched scenes. Mismatched predictions uniformly degraded VLM accuracy across all tasks, confirming that gains are strictly driven by scene-specific causal semantic cues.

Loss & Training¶

Beyond zero-shot benchmarking, the authors fine-tune Qwen3-VL-8B on the 18K FSU-QA training set using standard autoregressive cross-entropy loss over the answer tokens: $$\mathcal{L}_{\text{QA}} = -\sum_{i=1}^{L} \log P\left(y_i \mid y_{<i}, V_{-T_h:0}, \text{Traj}_{-T_h:0}, q\right)$$ Historical video sequences are sampled at 2 fps (7 frames across 3 seconds), and ego trajectories are encoded as textual $(x, y, \theta)$ coordinate sequences at 2 Hz.

Key Experimental Results¶

Main Results¶

FSU-Bench evaluates 13 leading foundation models (7 proprietary and 6 open-source) across all 9 tasks using only historical video frames and trajectory waypoints. Accuracies (%) are reported in the table below (source: original paper Table 2):

Model Group	Model	Rank	Overall	Speed	Turn	Lane	Rel. Dist.	Rel. Pos.	Ped. Int.	E-V Rel. Pos.	Risk Area	CFP
Closed-source	GPT-5	1	48.66	34.67	75.50	93.00	39.63	32.32	36.23	46.91	48.67	20.75
Closed-source	Claude-Sonnet-4.5	2	46.38	37.17	76.67	94.50	35.77	20.33	27.54	14.81	49.33	19.54
Closed-source	Claude-3.7-Sonnet	5	45.00	31.83	84.67	92.67	35.16	16.67	33.33	19.75	44.00	14.59
Closed-source	GPT-4o Mini	8	44.22	39.17	86.33	93.33	33.94	14.02	18.84	3.70	36.67	13.65
Closed-source	Gemini-2.5 Pro	9	44.04	37.83	66.17	91.67	35.37	22.76	30.34	37.04	25.33	18.47
Closed-source	Gemini-2.5 Flash	9	44.04	38.83	67.33	94.83	30.69	20.73	23.19	27.16	54.67	14.46
Closed-source	Gemini-2.0 Flash	13	41.43	32.00	74.17	91.17	33.54	15.04	30.43	7.41	29.33	12.45
Open-source	Qwen2.5-VL-72B	3	45.86	37.00	90.17	97.50	28.66	14.43	21.74	9.88	64.67	10.31
Open-source	Llama-4 Maverick	4	45.47	41.83	66.00	92.00	33.74	18.50	36.23	18.52	42.67	24.36
Open-source	Qwen2.5-VL-7B	6	44.74	34.00	88.33	97.50	27.85	14.43	37.68	3.70	63.33	8.43
Open-source	Qwen3-VL-32B	6	44.74	33.83	83.00	95.17	38.01	14.43	17.39	6.17	56.00	11.11
Open-source	Llama-4 Scout	11	41.95	41.67	70.33	81.50	30.69	14.43	36.23	3.70	50.67	16.06
Open-source	Qwen3-VL-8B (Zero-shot)	12	41.48	31.67	76.50	97.00	32.93	14.43	20.29	3.70	45.33	5.35
Fine-tuned	Qwen3-VL-8B-FI (Ours)	-	59.59	43.67	92.33	93.17	38.21	39.63	28.99	38.27	66.00	50.20

Ablation Study¶

Ablations examine how different modalities generated by Epona (diffusion + autoregressive) and DrivingWorld (pure autoregressive) contribute to foresight reasoning across VLMs, alongside shuffled control sanity checks (synthesized from Section 5.2 and Figures 5–6):

Predictive Input Configuration	Modality Mechanism	Task-specific Performance Profile	Relative Gain Trend	Control Validation
Baseline (History Only)	3s front video (7 frames) + history trajectory	Adequate static recognition; severely degraded counterfactual and multi-agent reasoning	Baseline reference (0.0)	-
+ Pred. Video Only	World model generated future video rollout	Significant boost on short-term dynamic and visual proximity tasks (Rel. Dist., E-V Rel. Pos.)	Moderate positive gain	-
+ Pred. Traj Only	World model predicted future spatial coordinate points	Distinct improvement in relational topology and agent intent (Ped. Intent, Risk Area)	Moderate positive gain	-
+ Pred. Video & Traj (Full Fusion)	Complementary visual motion cues and geometric structural constraints	Maximum accuracy across all tasks; DrivingWorld outperforms Epona in relational consistency	Strongest positive gain	Validated
Shuffled Control (Mismatched Scene WM)	Injected predicted video and trajectory from an unrelated scene	Injects contradictory spatio-temporal artifacts, severely degrading reasoning accuracy	Uniform drop below baseline	Confirms genuine semantic utility

Key Findings¶

Severe Deficit in High-Level Counterfactual Reasoning: While frontier models excel at short-term maneuver identification (e.g., Lane Change reaching 81%–97% and Turn Change reaching 66%–90%), performance collapses on Counterfactual Prediction (CFP). GPT-5 attains only 20.75%, while open-source models hover between 5% and 11%, exposing a fundamental inability in off-the-shelf VLMs to simulate hypothetical physical consequences.
Supervision via FSU-QA Closes the Foresight Gap: Supervised fine-tuning of Qwen3-VL-8B on FSU-QA boosts overall accuracy from 41.48% to 59.59%, with CFP leaping from 5.35% to 50.20%, demonstrating that targeted counterfactual data effectively unlocks latent causal foresight reasoning.
Architectural Discrepancy in World Models: DrivingWorld's purely autoregressive architecture consistently yields higher semantic utility than Epona's autoregressive-diffusion hybrid model on relational and interaction tasks, suggesting that discrete spatial auto-regression better enforces physical and geometric consistency over extended rollouts.

Highlights & Insights¶

Shifting Autonomous Driving VQA from Immediate Control to Long-Horizon Causal Simulation: Moving past reactive 1-second control prompts, the benchmark models 12-second evolutions and counterfactual safety outcomes, establishing a rigorous testbed for prospective physical reasoning.
Simulator-Free Geometric Counterfactual Ground Truth Construction: By combining closed-world 3D agent bounding boxes, kinematic trajectories, and HD-map topology, the authors formulate a scalable, physically grounded pipeline for labeling non-executed actions without simulator divergence.
VLM-as-a-Probe Evaluation for Generative World Models: Replacing superficial pixel-level similarity metrics (FID/FVD) with downstream VLM reasoning accuracy, complemented by shuffled negative controls, introduces a robust paradigm to audit the semantic realism of world models.

Limitations & Future Work¶

Single-View Front Camera Constraint: The dataset is currently limited to front-facing monocular video recordings, which omits blind spots and peripheral interactions crucial for full-scene intersection reasoning.
Cross-Domain Generalization: FSU-QA is built entirely within nuScenes (Boston and Singapore); validating model robustness on out-of-domain datasets like Waymo Open Dataset or nuPlan remains an open research avenue.
Loose Coupling of World Models and VLMs: The current framework treats world models as external generators; developing end-to-end architectures where world models serve as an internal mental simulation sandbox for VLMs is a promising next step.

vs NuScenes-QA & DriveLM: NuScenes-QA focuses on single-frame static queries, while DriveLM centers on immediate control choices. FSU-QA targets extended 12-second temporal horizons across 9 hierarchical cognitive tiers, prominently featuring counterfactual causal deduction.
vs OmniDrive: OmniDrive investigates counterfactual queries but evaluates output via geometric waypoint L2 displacement error. FSU-QA formulates the problem in natural language semantic space, diagnosing causal consequences like collision risk and spatial conflict.
vs Standard World Model Metrics (FID / FVD): Traditional metrics evaluate pixel distribution fidelity and temporal smoothness, which can be fooled by texture quality; FSU-Bench measures downstream reasoning utility via VLM probing, directly testing whether generated scenes obey physical causality.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneering formalization of Foresight Intelligence in driving VQA and introducing proxy VLM probing for world model semantic validation.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive benchmarking across 13 proprietary and open-source foundation models, two world models, and rigorous shuffled control ablations.]
Writing Quality: ⭐⭐⭐⭐⭐ [Well-structured narrative, lucid cognitive hierarchy taxonomy, and methodically sound empirical analyses.]
Value: ⭐⭐⭐⭐⭐ [Highly impactful for embodied AI safety, foundation model spatiotemporal reasoning, and world model verification.]