ScenarioControl: Vision-language Controllable Vectorized Latent Scenario Generation

Conference: ECCV 2026
Paper: ECCV Official
Project: https://light.princeton.edu/ScenarioControl
Area: Autonomous Driving
Keywords: Autonomous Driving Simulation, Vectorized Scenario Generation, Vision-Language Control, Latent Diffusion Model, Sensor Video Generation

TL;DR

ScenarioControl introduces the first vision-language controllable vectorized latent diffusion framework for 3D driving scenario synthesis, combining a cross-global control mechanism with collision regularization and cardinality regression to generate realistic 3D road layouts, reactive traffic agents, and sensor-level video continuations from text prompts or single forward-facing dashcam images.

Background & Motivation

The verification and safe deployment of autonomous driving stacks require rigorous evaluation on rare, safety-critical edge cases such as wrong-way drivers, unexpected road obstructions, or aggressive lane changes. However, such hazardous events rarely occur in real-world logs, making purely log-based data collection sample-inefficient. While conventional rule-based simulators like CARLA or MetaDrive provide parameterized environments, their handcrafted assets and scripted behaviors severely restrict perceptual and behavioral realism. Replay-based simulators, on the other hand, remain strictly tethered to recorded logs and cannot synthesize counterfactual situations or novel road topologies.

Recent generative simulation paradigms using diffusion models (e.g., SLEDGE, Scenario Dreamer) have enabled data-driven synthesis of vectorized or rasterized road layouts and initial agent placements. Nevertheless, an essential gap persists: scenario synthesis in these models is either unconditional—relying on unconstrained latent space sampling—or conditioned solely on existing structural layouts extracted from logged scenes. Users lack interpretable, intuitive control knobs to specify high-level scenario geometry, agent distributions, or environmental conditions through natural language prompts or single-frame dashcam observations. Meanwhile, video-space world models produce visual diversity but lack structured 3D topological representations directly usable by downstream motion planners.

Bridging dense multimodal conditioning features (natural language text embeddings or perspective dashcam images) and sparse vectorized scene tokens (lane centerlines and 3D bounding boxes) presents a severe architectural misalignment problem. Core idea: develop a cross-global control mechanism that couples fine-grained local cross-attention with a lightweight global latent context branch to inject text prompts or dashcam-style observations into a vectorized latent diffusion model, supplemented by a physical collision penalty and cardinality regression for consistent, controllable 3D scenario and sensor video generation.

Method

Overall Architecture

ScenarioControl operates over a \(64\text{m} \times 64\text{m}\) bird's-eye-view (BEV) scene representation augmented with vertical elevation \(z\) and height \(h\) to overcome flat-ground assumptions and facilitate downstream camera reprojections. The scene is formulated as a structured graph \(\mathcal{I} = \{\mathcal{O}, \mathcal{M}\}\) comprising 3D object bounding boxes \(\mathcal{O}\) and lane centerlines with connectivity \(\mathcal{M} = \{\mathcal{L}, \mathcal{A}\}\). The framework first trains a transformer-based vectorized autoencoder to encode scene elements into compact latent representations. During the generative phase, a latent diffusion model synthesizes lane and agent latents under the guidance of dense multimodal conditioning features (prompt embeddings or image features with estimated depth) via the cross-global control module. Decoded scenes are then spatially outpainted, dynamically simulated via behavior policies, and reprojected as 3D wireframe sequences to condition a video diffusion model for photorealistic sensor rendering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Control Inputs<br/>Text prompt or single dashcam image + depth"] --> B["Cross-Global Control Mechanism<br/>Local cross-attention + latent global-context branch"]
    B --> C["Vectorized Latent Diffusion Denoising<br/>Joint lane & agent diffusion + collision penalty"]
    C --> D["Element Cardinality Predictor<br/>Attention-based regression of agent & lane counts"]
    D --> E["Vectorized Scene Decoding & Outpainting<br/>3D bounding boxes with elevation & lane graphs"]
    E --> F["Behavior Simulation & Video Rendering<br/>Agent trajectory rollouts + BEV2Cam wireframe video DiT"]

Key Designs

1. Cross-Global Control Mechanism: Fusing Sparse Scene Tokens with Dense Conditioning Streams Directly attending sparse vectorized tokens to dense visual features or prompt tokens using standard cross-attention lacks inductive bias and leads to sample inefficiency when capturing macro-level road structure. To address this, the cross-global control mechanism introduces two parallel branches that share the key/value linear projections of the conditioning stream. Given scene queries \(Q \in \mathbb{R}^{N_q \times d}\) and conditioning tokens \(F \in \mathbb{R}^{N_k \times d}\), the first branch computes standard local cross-attention via FlashAttention \(Y_1 = \mathrm{Attn}(Q, K(F), V(F))\). In parallel, a set of learned latent tokens \(T \in \mathbb{R}^{L \times d}\) summarizes the conditioning tokens into a compact global context \(\tilde{T} = \mathrm{Attn}(T, K(F), V(F))\), which is subsequently queried by the scene tokens \(Y_2 = \mathrm{Attn}(Q, K(\tilde{T}), V(\tilde{T}))\). The two pathways are combined via an adaptive gate parameter \(g\): $\(Y = Y_1 + \tanh(g) Y_2\)$ The gate is initialized to \(g = 0\) (so \(\tanh(g) \approx 0\)), allowing the model to start as standard cross-attention and progressively incorporate global scene layout cues with negligible parameter overhead.

2. Physical Consistency Collision Penalty: Suppressing Spatial Overlap under Ambiguity Under-specified natural language prompts or visual occlusions in single-view camera images can lead the diffusion model to place multiple candidate vehicles into the same plausible spatial coordinates, causing severe overlapping bounding boxes in decoded layouts. To enforce geometric plausibility without sacrificing generative diversity, a differentiable collision regularization loss \(\mathcal{L}_{\text{col}}\) is introduced during reverse diffusion. At selected denoising timesteps \(\tau\), intermediate latents are decoded into agent bounding boxes, and pairwise spatial overlap areas \(\text{overlap}(i, j)\) are penalized: $\(\mathcal{L}_{\text{col}} = \frac{1}{N} \sum_{i \neq j} \tanh\left(\frac{\text{overlap}(i, j)}{\zeta}\right)\)$ The penalty is modulated by a timestep-dependent weighting factor \(w_\tau = 1 - \sqrt{1 - \bar{\alpha}_\tau}\), which downweights early diffusion steps when noise dominates and emphasizes late diffusion steps where bounding box geometry is structurally reliable.

3. Cardinality Prediction and Canonical Layout Outpainting: Long-Horizon Spatial Extension Because graph-based scenes contain variable numbers of lane polylines and agents, a lightweight attention-based regression module \(f_{\text{count}}\) is trained to predict the appropriate cardinalities \((N_o, N_l)\) directly from the conditioning features \(F\). Furthermore, to accommodate the observational asymmetry between forward-facing cameras and omnidirectional text descriptions, the model supports two canonical crops: an ego-centered crop \(F_P\) and a forward-shifted crop \(F_I\). During inference, large-scale driving environments are synthesized through masked denoising outpainting, wherein observed graph latents are held clamped while out-of-view tokens are sampled conditionally, providing extended continuous tracks for multi-agent behavior simulation.

Loss & Training

The framework follows a two-stage training paradigm. First, the transformer-based autoencoder encoder \(f_e\) and decoder \(f_d\) are trained to reconstruct 3D vectorized scene elements. Second, the autoencoder weights are frozen, and the latent diffusion backbone and conditioning modules are trained using the standard \(\epsilon\)-prediction objective: $\(\mathcal{L}_{\text{LDM}} = \mathbb{E}_{\tau, \epsilon \sim \mathcal{N}(0, \mathbf{I}), \mathcal{C}} \left[ \|\epsilon - \epsilon_\theta(\mathbf{Z}_\tau, \tau; \mathcal{C})\|^2 \right]\)$ The final objective combines diffusion and collision regularization: \(\mathcal{L} = \mathcal{L}_{\text{LDM}} + \lambda_{\text{col}} w_\tau \mathcal{L}_{\text{col}}\). Conditioning inputs (text tokens encoded by a frozen language model or image tokens encoded by a frozen vision backbone concatenated with monocular depth maps and sinusoidal positional embeddings) are randomly dropped with probability \(p_{\text{CFG}}\) to enable Classifier-Free Guidance at inference time.

Key Experimental Results

Main Results

On the nuPlan benchmark, ScenarioControl is compared against the prior state-of-the-art vectorized scene generation model (Scenario Dreamer) across control adherence metrics—Cosine Control Similarity (CCS), Shuffled Perturbation Gap (SPG), and Control Sensitivity Correlation (CSC)—and topological realism metrics—Fréchet Distance (FD), Connectivity (Conn.), Reach, and Convenience (Conve.).

Setting	Method	Global CSC ↑	Lane CSC ↑	Agent CSC ↑	Realism FD ↓	Lane Conn. ↓	Lane Reach ↓
Prompt Control (\(C_P\))	Scenario Dreamer	0.3510	0.2885	0.0625	2.44	0.22	0.12
Prompt Control (\(C_P\))	ScenarioControl (Ours)	0.3809	0.4006	0.1626	0.51	0.32	0.11
Image Control (\(C_I\))	Scenario Dreamer	0.4320	0.3166	0.0623	9.94	4.19	1.01
Image Control (\(C_I\))	ScenarioControl (Ours)	0.6274	0.6100	0.2255	2.26	3.79	0.95

Ablation Study

Ablations on different conditioning attention designs (Table 3) and the collision loss / cardinality prediction components (Table 4) validate the core architectural choices.

Table 1: Comparison of conditioning attention mechanisms under image conditioning (\(C_I\)) | Attention Mechanism | Global CSC ↑ | Lane CSC ↑ | Agent CSC ↑ | Agent AP ↑ | Collision Rate (%) ↓ | |---|---|---|---|---|---| | Concatenation | 0.4105 | 0.4094 | 0.1314 | 23.37 | 30.18 | | Full Cross-Attention | 0.6137 | 0.6075 | 0.2195 | 32.00 | 16.07 | | Gated Attention | 0.4637 | 0.4953 | 0.1514 | 25.13 | 14.92 | | Linear Attention | 0.5299 | 0.5861 | 0.2012 | 27.20 | 16.92 | | Cross-Global Control (Ours) | 0.6274 | 0.6100 | 0.2255 | 33.46 | 17.07 |

Table 2: Ablation of collision penalty \(\mathcal{L}_{\text{col}}\) and cardinality regressor \(f_{\text{count}}\) | Collision Loss \(\mathcal{L}_{\text{col}}\) | Cardinality Regressor \(f_{\text{count}}\) | Collision Rate (%) ↓ | Agent AP (%) ↑ | |---|---|---|---| | ✗ | ✗ | 19.81 | 39.55 | | ✓ | ✗ | 14.07 | 38.99 | | ✗ | ✓ | 18.78 | 38.70 | | ✓ | ✓ | 13.36 | 38.03 |

Key Findings

• The cross-global control module outperforms simple concatenation by 52.8% on global CSC and reduces the collision rate from 30.18% to 17.07%. It also outperforms standard full cross-attention by 4.6% relative gain in Agent AP (33.46% vs. 32.00%), confirming that learned global latent tokens provide vital inductive bias for scene-level layout cohesion.
• Incorporating the differentiable collision loss \(\mathcal{L}_{\text{col}}\) consistently cuts vehicle collisions by 5.5% to 6.4%, effectively mitigating overlapping box artifacts in occluded or under-specified scene regions.
• When transferred zero-shot to the Waymo Open Motion dataset, ScenarioControl maintains superior prompt controllability over a cascaded VLM+TrafficGen baseline (which requires 13.05 s/sample due to slow GPT-4o-mini map generation), achieving a Lane CSC of 0.5179 versus 0.0585 for the baseline.

Highlights & Insights

• Unified Dense-to-Sparse Conditioning: Unlike prior works that rely on intermediate raster representations or cascaded LLM tool-calling to build lane graphs, ScenarioControl directly guides 3D sparse vector tokens from dense visual/text embeddings in latent diffusion space.
• Elevation-Aware Structured Synthesis: Integrating vertical elevation \(z\) and height \(h\) directly into the vectorized scene graph eliminates the planar ground assumption, allowing generated 3D boxes and lane lines to be reprojected into camera wireframes for photorealistic video rendering via LoRA-adapted Wan 2.2.
• Large-Scale Benchmark Creation: The authors curated 500K paired VLM captions for nuPlan driving scenes, aligning high-level natural language descriptions directly with ground-truth vectorized HD map topologies.

Limitations & Future Work

• Single-View Visual Hallucination: Conditioned on a single forward camera frame, outpainting unobserved side and rear regions relies heavily on learned dataset priors, which can introduce plausible yet counterfactual topologies in complex intersections.
• External Trajectory Coupling: While the initial 3D scene and spatial outpainting are generated end-to-end, multi-agent temporal rollouts still rely on external behavior simulation models rather than a fully unified spatio-temporal latent world model.

Related Work & Insights

• vs. Scenario Dreamer: While both model vectorized driving environments using latent diffusion, Scenario Dreamer is unconditional or limited to simple layout priors; ScenarioControl introduces vision-language control, improving topological fidelity (FD from 9.94 to 2.26) and enabling targeted prompt-guided scenario creation.
• vs. SLEDGE & TrafficGen: SLEDGE relies on raster-to-vector autoencoders and rule-based traffic models, while TrafficGen assumes pre-existing road graphs; ScenarioControl jointly synthesizes both the road topology and agent distributions conditioned directly on multimodal inputs.
• vs. Video World Models (GAIA-1/2, Cosmos-Drive-Dreams): Pixel-space generative models lack explicit, editable 3D vectorized representations for closed-loop motion planners; ScenarioControl generates structured graphs that serve both classical planning stacks and sensor video rendering.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneering integration of vision-language conditioning into 3D vectorized latent scenario diffusion via cross-global attention]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive quantitative evaluation on nuPlan, zero-shot transfer to Waymo, multi-level control adherence metrics, and video rendering demonstrations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear formulation, well-motivated architectural diagrams, and rigorous ablation analyses]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical bridge between natural language descriptions, 3D autonomous driving simulators, and sensor-realistic video generation]