FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation

Conference: ECCV 2026
arXiv: 2606.20110
Code: None (the paper mentions supplementary material, but no repository link is provided)
Area: Diffusion Models / Autonomous Driving / Controllable Generation / Data Augmentation
Keywords: Frozen Diffusion Models, Zero-Shot Text-Guidance, Multi-View Consistency, Driving Scene Generation, Long-Tail Data Augmentation

TL;DR

FrozenDrive completely freezes the pretrained Stable Diffusion backbone with zero added parameters, achieving both multi-view and temporal consistency purely by rearranging the inputs of self-attention (concatenating multi-view features and injecting reference frames). This preserves the original diffusion prior to enable zero-shot text-guided driving scene generation. Directed synthesis of rare adverse weather data (e.g., night, rain, snow) via text prompts significantly enhances downstream perception and planning robustness.

Background & Motivation

Background & Limitations of Prior Work: Autonomous driving data collection is expensive, and adverse weather or long-tail rare events are almost impossible to collect in bulk. Consequently, real-world datasets are heavily biased toward sunny daytime scenarios, making models highly vulnerable to long-tail conditions. Recently, the mainstream approach of driving scene synthesis utilizes diffusion models: using BEV layouts, 3D boxes, and occupancy maps as 2D spatial conditions (mostly injected via ControlNet) to adapt pretrained models like SD / SVD / DiT to driving datasets. However, to support multi-camera rigs and temporal dynamics, these methods either generate each view almost independently with weak cross-view sharing, or introduce and fine-tune learnable cross-view/temporal attention layers in the pretrained backbone. The problem is that both strategies require extensive fine-tuning of the backbone, and since the dataset itself has biased coverage (lacking sufficient samples of snow, night, and rain), the rich priors in the pretrained model regarding adverse/unseen scenarios are overwritten or catastrophically forgotten. Text-to-image alignment is weakened, causing the model to only perform local, surface-level edits on "atmospheres seen during training." When prompted to synthesize unseen heavy snow, the generation fails. Furthermore, updating the entire backbone sacrifices per-object fidelity, especially for rare classes.

Key Challenge & Goal: There is a fundamental conflict here: achieving multi-view/temporal consistency requires adding components and training the backbone; however, training the backbone erodes the pre-trained priors and destroys the zero-shot text generation capability. The goal of this work is to achieve robust consistency while fully preserving the knowledge of the pretrained diffusion model (especially text-image alignment) so that synthesizing scenes using text descriptions unseen during training remains feasible. The authors examine the three mainstream adaptation strategies (full/partial fine-tuning, adding trainable adapters, ControlNet conditioning) under the metric of "knowledge preservation." They make a key empirical observation: even adding a tiny amount of trainable parameters (multi-view or temporal cross-attention) to a frozen backbone induces a similar prior drift, narrowing visual diversity and degrading prompt fidelity.

Key Insight / Core Idea: The core idea of this work is: do not add any parameters to the backbone or update any weights; consistency is achieved entirely by "modifying the input seen by self-attention." Specifically, the inputs to the self-attention layers are rearranged: for multiple views, the latents from all cameras are concatenated for a single attention pass (circular cross-view alignment); for temporal consistency, keys and values from the previous frame are concatenated into the current frame's memory bank (temporal coherence). The attention weights themselves remain fixed as frozen LDM parameters. By keeping the backbone frozen and parameter-free, the pretrained text-image alignment is fully preserved, enabling zero-shot text guidance. This is coupled with an object-presence ratio loss to recover fidelity for rare objects.

Method

Overall Architecture

The skeleton of FrozenDrive is a multi-view generation pipeline conditioned on pure ControlNet conditioning + frozen SD backbone. The inputs consist of five conditional signals: (i) scene layout (HD map layers + 3D bounding boxes projected onto multi-channel masks of each view), (ii) depth maps (obtained by ray-casting 3D occupancy aggregated from LiDAR), (iii) per-view camera indicators (view indices encoded through Fourier features), (iv) relative pose to the previous frame (constructed as pixel-wise 2D spatial correspondence maps), and (v) text descriptions. The first four signals are multi-view and pixel/space-aligned; they are processed by a lightweight embedding network and added to the latent to be fed into ControlNet. The text is separately encoded by a frozen CLIP text encoder and processed via cross-attention. The control features generated by ControlNet modulate the frozen diffusion backbone without altering or adding parameters to it.

Crucially, while the backbone is frozen, the authors replace the original self-attention in the LDM decoder with "Knowledge-Preserving Spatiotemporal Attention"—not with a new module, but by keeping the projection weights (\(\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V\) fully frozen) and changing "which features are fed in to attend to each other." This is built on two complementary mechanisms: Multi-View Inflated Self-Attention (MISA) for cross-view consistency, and Temporal Reference Self-Attention (TRSA) for inter-frame consistency, both operating within the same frozen block. Finally, an object-presence ratio loss is overlaid to bias learning toward rare classes. The entire pipeline produces view-consistent and temporally stable multi-view scenes in a single forward pass from "text + driving stack signals."

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Driving Stack Signals<br/>HD Map/Depth/View/Pose"] --> B["Frozen SD Backbone<br/>+ ControlNet Conditioning"]
    T["Text Prompt"] -->|Frozen CLIP Encoding| B
    B --> C["Knowledge-Preserving Spatiotemporal Attn<br/>(Replaces Decoder Self-Attn)"]
    subgraph SG["Projection Weights Frozen Throughout"]
    direction TB
    C --> D["Multi-View Inflated Self-Attention<br/>Concat All View Latents"]
    C --> E["Temporal Reference Self-Attention<br/>Concat Prev Frame K/V"]
    end
    D --> F["Object-Presence Ratio Loss<br/>Weight Rare Classes"]
    E --> F
    F --> G["Multi-View & Temporally Consistent Scene"]

Key Designs

1. Knowledge Preservation Principle: Frozen Backbone and Zero New Parameters, Achieving Consistency via "Rearranging Inputs" Rather Than "Adding Layers"

This principle explicitly addresses the core conflict mentioned earlier—fine-tuning added layers erodes priors. The empirical finding is key: not only do full/partially fine-tuned backbones induce drift, but even adding a tiny amount of trainable multi-view/temporal cross-attention layers on top of a frozen backbone triggers the same prior drift (narrowing diversity and weakening prompt fidelity, as seen in the "SD + CA" baseline of the ablation study). Thus, the design principle is straightforward: the backbone is completely frozen with absolutely zero new parameters introduced, and consistency is achieved entirely by modifying the input context of the self-attention. The only trainable components are the ControlNet and conditional embedders, which are initialized from scratch (only 0.37B out of the 1.40B total parameters are trainable). Since the backbone and its text-image alignment remain untouched, the model retains zero-shot text guidance for weather/scene combinations unseen in the training set—which is why it can synthesize blizzards with "Snowy weather. Heavy snow." while fine-tuning methods fail.

2. Multi-View Inflated Self-Attention (MISA): Concatenating All Camera Latents into a Single Sequence for a Frozen Attention Pass

To ensure cross-view consistency (overlapping views must be consistent in geometry, appearance, and semantics), the standard practice is to add cross-view attention layers, which require training. MISA's approach is to directly concatenate the \(n_{\text{view}}\) camera latent tokens into a joint sequence and feed them into the same frozen self-attention: let \(\mathbf{X}^{(v)}\in\mathbb{R}^{N_v\times d_{\text{model}}}\) be the feature token of view \(v\). They are concatenated into \(\tilde{\mathbf{X}}:=[\mathbf{X}^{(1)};\mathbf{X}^{(2)};\cdots;\mathbf{X}^{(n_{\text{view}})}]\), and then:

\[\mathrm{Attn}(\tilde{\mathbf{X}})=\mathrm{softmax}\!\left(\frac{(\tilde{\mathbf{X}}\mathbf{W}^Q)(\tilde{\mathbf{X}}\mathbf{W}^K)^\top}{\sqrt{d_h}}\right)(\tilde{\mathbf{X}}\mathbf{W}^V)\]

where \(\mathbf{W}^Q,\mathbf{W}^K,\mathbf{W}^V\) are the frozen LDM parameters, and \(d_h\) is the dimension per head. The attention, originally computed within each view, is "inflated" to allow cross-view interactions without changing any weights. To let the attention know which view each token comes from, a lightweight view embedding \(\mathbf{e}_{\text{view}}\) is used: view indices (front-left, front-center, front-right, back-right, back-center, back-left, mapped to 0–5) are encoded via Fourier features and fed into ControlNet as extra conditions, providing view identity and adjacency structure to improve cross-view correspondence at near-zero cost.

3. Temporal Reference Self-Attention (TRSA): Concatenating Key/Value of the Previous Frame into the Current Frame's Attention Bank

To ensure temporal consistency (continuous frames should not have hallucinatory mutations under ego/object motions), TRSA follows the same logic as MISA: all LDM attention weights remain frozen, and only the input context is extended, reusing the previous frame as an explicit reference. For the current frame \(i\) and reference frame \(i-j\), the latents of the same noise level are retrieved. The current \(\mathbf{Q}_i,\mathbf{K}_i,\mathbf{V}_i\), and reference \(\mathbf{K}_{i-j},\mathbf{V}_{i-j}\) are computed using frozen weights. The reference keys/values are then appended to the current frame's bank:

\[\tilde{\mathbf{K}}_i=[\mathbf{K}_i;\mathbf{K}_{i-j}],\quad \tilde{\mathbf{V}}_i=[\mathbf{V}_i;\mathbf{V}_{i-j}],\quad \mathrm{Attn}(\mathbf{X}_i)=\mathrm{softmax}\!\left(\frac{\mathbf{Q}_i\tilde{\mathbf{K}}_i^\top}{\sqrt{d_h}}\right)\tilde{\mathbf{V}}_i\]

This allows the current frame to retrieve temporally aligned appearance and geometric cues (objects, weather) from the previous frame, stabilizing rollouts without introducing new parameters (at inference, \(j{=}1\) is used to reference the most recently generated frame). Frame-to-frame motion is guided by a lightweight relative pose embedding \(\mathbf{e}_{\text{pose}}\): instead of directly injecting 6-DoF transforms, each pixel \((x,y)\) is unprojected to \((x,y,0)\), transformed using the known 3D relative pose, projected back to planar coordinates \((x',y')\), and Fourier-encoded to construct a dense correspondence map injected via ControlNet. MISA and TRSA are organically integrated into the same frozen block and only applied in the LDM decoder (as the decoder aligns better with the scene-level conditions of ControlNet), jointly producing view-consistent and temporally stable results.

4. Object-Presence Ratio Loss: Weighting Pixels by Class Scarcity to Shift Learning to the Long-Tail

Supervised driving datasets suffer from severe long-tail distributions (e.g., in nuScenes, "bicycle" occurs only ~2.3% as frequently as "car"). Since rendering fidelity naturally correlates with observation frequency, rare classes suffer. On top of the standard DDPM noise prediction loss \(\mathcal{L}=\mathbb{E}[\|\epsilon-\epsilon_\theta(\cdot)\|^2]\), the authors construct a pixel-wise weight map to scale up the loss for pixels belonging to rare classes:

\[\mathcal{L}_{\text{total}}=\frac{1}{|\Omega|}\sum_{p\in\Omega}\bigl(1+\lambda\,w(p)\bigr)\,\mathcal{L}(p),\qquad w(p)=\max_{k\in\mathcal{K}_p}w_k,\ \ \mathcal{K}_p=\{k\mid p\in\mathcal{M}_k\}\]

where \(\Omega\) is the image grid, \(\mathcal{M}_k\) is the foreground mask obtained by projecting the 3D bounding box of class \(k\), and \(w_k\) is the class-specific weight (larger for rarer classes, specifically set as \(w_k=(o_t/o_k)\) where \(o_t/o_k\) is the ratio of total objects to the number of objects of this class); pixel weights without objects are set to 0. The max-overlap rule allows the rarest instance in overlapping regions to dominate the weight (e.g., if a pixel belongs to both a car and a bus, with \(w_{\text{car}}{=}2.29\) and \(w_{\text{bus}}{=}71.78\), it takes 71.78), with \(\lambda\) set to 0.02. This explicitly biases learning toward under-represented categories instead of favoring frequent classes. ⚠️ The precise form of \(w_k\) (whether it carries a \(p\) subscript) is subject to Eq.(H) in the original paper.

Loss & Training

Training only updates the ControlNet and conditional embedders, while the backbone (Stable Diffusion v1.5) remains completely frozen without any added layers. The optimizer is AdamW with a learning rate of \(1\times10^{-4}\), training for 200K iterations with a batch size of 4, across a two-stage resolution: first 150K iterations at \(224\times400\), then 50K iterations at \(448\times800\). During training, MISA is enabled to learn multi-view consistency, while during inference, TRSA is additionally activated (\(j{=}1\)) to generate temporally consistent sequences. The final frames are generated at \(448\times800\) and then bilinearly upsampled to the original nuScenes resolution of \(900\times1600\) for downstream tasks like UniAD / SparseDrive. All training is completed on 2× A100 GPUs. Inference requires only a single A100 GPU with 12.43GB VRAM, taking 0.97s per frame/step (compared to MagicDrive-V2 which requires 4× A100 GPUs and 59.95GB per GPU).

Key Experimental Results

Main Results

Multi-view images are generated under the nuScenes validation set conditions, and evaluated using UniAD for perception/planning and FVD for generation quality. FrozenDrive achieves the best downstream performance among SD-based methods, particularly in BEV segmentation (indicating strong spatial multi-view consistency). The FVD is the lowest among SD-based image diffusion baselines (STDiT video diffusion models have a lower FVD due to explicit temporal modeling, which is expected).

Method	Backbone	3DOD mAP↑	NDS↑	BEV mIoU (Drivable)↑	L2 Avg↓	FVD↓
nuScenes (Real)	-	37.98	49.85	69.14	1.05	-
MagicDrive (ICLR'24)	SD	12.92	28.36	51.46	1.22	218.1
Panacea (CVPR'24)	SD	13.72	27.73	52.37	1.23	139.0
DriveArena (ICCV'25)	SD	16.06	30.03	59.37	1.18	185.3
MagicDrive-V2 (ICCV'25)	STDiT	15.24	31.25	58.63	1.09	81.6
X-Scene (NeurIPS'25)	SD	20.40	31.76	61.96	1.15	179.7
DiST-4D (ICCV'25)	STDiT	15.63	32.44	60.32	1.19	22.6
FrozenDrive	SD	21.87	35.32	64.27	1.05	136.8

Downstream data augmentation (using FrozenDrive's pure text prompts to synthesize targeted night/rain samples to train SparseDrive) serves as even more convincing evidence:

Condition	Method	3DOD mAP↑	NDS↑	Online Mapping mAP↑	L2 Avg↓
Night	Baseline (Normal weather only)	6.62	15.19	5.99	1.40
Night	DriveArena	8.89	17.52	7.00	1.09
Night	MagicDrive-V2	12.68	17.32	11.69	1.11
Night	FrozenDrive	18.15	24.95	21.03	0.93
Rain	Baseline	31.60	41.20	24.75	0.75
Rain	MagicDrive-V2	33.93	42.20	30.02	0.73
Rain	FrozenDrive	35.15	44.26	31.39	0.58

The performance gain at night is particularly substantial (with online mapping mAP scaling up from 5.99 on the baseline to 21.03, and planning L2 dropping from 1.40 to 0.93). The authors attribute this to FrozenDrive's ability to preserve the scene geometry of the original layouts while realistically rendering low-light appearances. Synthesizing night scenes under this framework is far closer to real night shots than using rule-based filters, DriveArena, or MagicDrive-V2.

Ablation Study

Spatiotemporal Attention (MISA + TRSA complementarity):

Configuration	BEV Drivable mIoU↑	BEV Crossing↑	FVD↓	Explanation
MISA + TRSA (Full)	56.5	8.9	144.1	Full model, performs well on both metrics
MISA only (w/o TRSA)	55.1 (-1.4)	7.8 (-1.1)	174.2 (+30.1)	Temporal quality drops, FVD worsens, BEV remains acceptable
TRSA only (w/o MISA)	51.9 (-4.6)	5.6 (-3.3)	145.1 (+1.0)	FVD slightly better, but BEV severely degrades (cross-view consistency lost)

Object-Presence Ratio Loss (sorted from high to low frequency):

Configuration	mAP↑	Car AP↑	Motorcycle AP↑	Bicycle AP↑
w/ loss	21.9	38.6	10.0	9.2
w/o loss	16.1 (-5.8)	33.5 (-5.1)	0.4 (-9.6)	1.9 (-7.3)

Key Findings

• MISA handles space and TRSA handles time, making them highly complementary and indispensable: Removing TRSA mainly degrades FVD (+30.1) and temporal consistency. Removing MISA causes BEV segmentation to completely collapse (Drivable -4.6, Crossing -3.3). Only when both are used together do we achieve the highest BEV mIoU and the lowest FVD.
• The object-presence ratio loss acts as a life-saver for rare classes: Motorcycle AP increases from 0.4 to 10.0 and Bicycle AP from 1.9 to 9.2 (where these classes only account for about 1.07% and 1.00% of the training set, respectively), with an overall +5.8 increase in mAP. The high-frequency class (car) also gains +5.1, demonstrating that re-weighting does not sacrifice performance on common classes.
• Knowledge forgetting is quantifiable: When MISA/TRSA are replaced by learnable cross-attention ("SD + CA"), even though multi-view/temporal consistency is maintained, the model visibly under-reflects text under unseen prompts like "Snowy weather. Heavy snow." with very weak snow cues. In out-of-distribution (OOD) snow conditions (which are completely absent from the training set), FrozenDrive achieves a CLIP score of 0.2507, significantly outperforming DriveArena (0.2277) and MagicDrive-V2 (0.2181). This highlights the unique advantage of the frozen design for OOD generalization.
• Direct consistency measurements remain robust: The lane boundary alignment mIoU of 33.58 matches that of "SD + CA" (33.55) which requires extra learnable cross-attention + partial fine-tuning. This proves that high geometric precision can be obtained even with zero new parameters. VBench temporal consistency is slightly inferior to MagicDrive-V2 (since the latter is a video diffusion model) but outperforms fine-tuning baselines such as SD+CA and DriveArena.

Highlights & Insights

• The counter-intuitive strategy of "no training is better than training": The core contribution is not what module was added, but what was left out. Multi-view/temporal consistency is achieved entirely by rearranging the input of the self-attention (concatenating views, concatenating previous frame K/Vs), while the attention weights remain completely untouched. This fully preserves the pretrained text-image alignment, enabling zero-shot text guidance. This concept of "reshaping inputs instead of adding parameters" can be directly applied to any frozen foundation model scenarios where one wants to inject capabilities without ruining existing priors.
• Identifying "adding small parameters also causes forgetting" as an independent finding: The paper does not stop at the consensus that "full fine-tuning causes forgetting." It empirically shows that adding even a few learnable cross-attention layers on top of a frozen backbone triggers prior drift (revealed by the failed snow rendering of SD+CA and collapsed CLIP scores). This provides hard evidence for the necessity of "parameter-free" adaptation, making the argument much stronger.
• The max-overlap weighting is a clever, simple design: Having the rarest instance dominate overlapping regions prevents car, a high-frequency class taking up a large area, from drowning out bicycle/motorcycle in the pixel-level loss. This is a highly reusable weighting trick for long-tail generation.
• Data augmentation metrics provide much harder evidence than FVD: The authors purposely evaluate the value of their model based on "downstream AD model perception/planning improvement on night/rain" rather than simply comparing CLIP/FVD score plausibility. A boost of this magnitude (online mapping mAP from 5.99 to 21.03 at night) is far more indicative of the practical utility of synthetic data than simple generation quality numbers.

Limitations & Future Work

• Long-range temporal generation is still inferior to video diffusion: The authors acknowledge that the parameter-free frozen strategy trails behind recent video diffusion models in long-range temporal coherence (as evidenced by DiST-4D's FVD of 22.6 and MagicDrive-V2's higher VBench temporal score). It essentially relies on local temporality by "concatenating the previous frame" rather than true video modeling.
• Lack of direct evaluation protocols for data augmentation quality: The existing protocol cannot directly evaluate the quality of the "synthetic augmentation data" itself, relying instead on indirect downstream task results. The authors list "designing more faithful augmentation quality metrics" as a future direction.
• Tied to SD v1.5: The pipeline was only validated on Stable Diffusion. Extending it to stronger video diffusion or DiT backbones remains a clear open direction. Whether similar consistency can be achieved purely by "rearranging inputs" in DiT attention structures has yet to be tested.
• Self-assessment additions: The data augmentation experiments only cover two nuScenes categories (night/rain, while snow is only qualitatively/CLIP evaluated without downstream numbers) with only 12 evaluation scenes per domain, which is relatively small. The authors also point out that CLIP scores only reflect stylistic plausibility rather than realness, meaning conclusions rely heavily on downstream metrics.

Related Work & Insights

• vs MagicDrive / Panacea / DriveArena (SD-based, fine-tuning backbone or adding layers): These methods achieve consistency by adding learnable cross-view/temporal attention layers on top of a pretrained backbone and fine-tuning it. In contrast, this paper completely freezes the backbone and purely rearranges the attention inputs. The key difference is that this work preserves the zero-shot text capability (enabling the synthesis of unseen blizzards), which other methods fail to do or only manage via local/surface-level editing under unseen weather, although the trade-off is slightly weaker long-range temporal consistency.
• vs MagicDrive-V2 / DrivingSphere / DiST-4D (DiT/STDiT video diffusion): These methods explicitly model temporal dynamics using video diffusion, yielding lower FVD and stronger long-term temporal consistency. However, they require massive VRAM and multi-GPU setups (MagicDrive-V2 requires 4× A100 GPUs, 60GB VRAM per card) and suffer from prior loss under training set coverage bias. This work runs inference on a single 12GB GPU and outperforms them in downstream data augmentation (especially at night), indicating that "preserving priors" is more valuable than "stronger temporal modeling" for data augmentation.
• vs ControlNet + Full Fine-tuning paradigm: This paper inherits the concept of using ControlNet to inject structural conditions, but insists on training only the ControlNet and embedder, leaving the backbone untouched. This contrasts with the typical practice of co-fine-tuning ControlNet and the backbone. The core insight is that ControlNet is sufficient to supply spatial control, and the backbone's sole job should be preserving generative priors.

Rating

• Novelty: ⭐⭐⭐⭐⭐ "Parameter-free frozen backbone + pure input rearrangement for multi-view/temporal consistency + preserving zero-shot text guidance" is a clean and counter-intuitive combination of innovations. Both design motivation and empirical results are highly solid.
• Experimental Thoroughness: ⭐⭐⭐⭐ Generation quality, downstream augmentation, double ablations, dedicated knowledge-forgetting study, and direct consistency evaluations are all present. However, downstream augmentation was only evaluated on night and rain, with only 12 scenes per domain, and snow lacks downstream results.
• Writing Quality: ⭐⭐⭐⭐ The chain of motivation (conflict \(\rightarrow\) principle \(\rightarrow\) mechanism) is clear, and MISA, TRSA, and losses are well-explained. Some mathematical notations (e.g., precise form of \(w_k\), some table parameters) require cross-checking with the supplementary materials.
• Value: ⭐⭐⭐⭐⭐ Running on a single GPU, generating targeted long-tail adverse weather purely from text, and demonstrating huge downstream gains in night scenarios make this work highly valuable for real-world autonomous driving perception and planning.

Related Papers

• [ECCV 2026] DriveVA: Video Action Models are Zero-Shot Drivers
• [CVPR 2025] Zero-Shot 4D Lidar Panoptic Segmentation
• [ECCV 2026] LaGen: Towards Autoregressive LiDAR Scene Generation
• [CVPR 2026] DrivePTS: A Progressive Learning Framework with Textual and Structural Enhancement for Driving Scene Generation
• [ECCV 2024] Rethinking Data Augmentation for Robust LiDAR Semantic Segmentation in Adverse Weather