DiverseAD: A Large-Scale Driving Dataset with Diverse Atmospheric Conditions¶

Conference: ECCV2026
Paper: Official ECCV page
PDF: Full paper
Authors: Haoyu Wang, Baorui Ma, Donglin Di, Suhang Xuan, Hao Li, Shiliang Zhang
Area: Autonomous Driving
Keywords: weather robustness, end-to-end planning, driving intent, scene structure, driving dataset

TL;DR¶

DiverseAD provides both a 150K-scene driving dataset covering diverse atmospheric conditions and a dual-anchor feature-learning method: trajectory direction patterns and relationships from frozen DINOv3 features guide visual representations, reducing average L2 trajectory error on DiverseAD from 1.44 m for fine-tuned FSDrive to 1.24 m.

Background & Motivation¶

End-to-end autonomous driving can predict future trajectories directly from surround-view images, motion history, and navigation commands, but visual regularities in training data are not necessarily reliable driving cues. The same road changes in color, reflections, and visibility under fog, strong illumination, or darkness. With trajectory supervision alone, a model may associate incidental appearances with driving actions. Existing multimodal driving datasets have advanced planning research, while some adverse-weather datasets cover perception tasks. The authors argue that sufficiently large datasets combining atmospheric diversity with ego-trajectory annotations remain lacking.

The paper therefore addresses data coverage and training constraints together rather than merely replacing a network component. Its data collection captures variations in weather, road types, and driving behavior, with perception, trajectory, and question-answer annotations. Its model explicitly specifies which samples should receive similar visual representations. More weather examples alone do not ensure that the model ignores irrelevant appearance, while indiscriminate invariance can discard road geometry needed for planning. Task-relevant references are therefore necessary.

The authors choose two references: driving intent derived from ground-truth trajectories and scene structure supplied by a pretrained vision model. Here, invariance is an encouraged property of the representation, not a strict guarantee covering every atmospheric condition or driving decision. Core Idea: on diverse real-world driving data, jointly constrain visual features through trajectory-direction relationships and scene-structure relationships, reducing reliance on atmospheric appearance while retaining action and road-layout information.

Method¶

Overall Architecture¶

The work has three key designs: diverse data construction, driving-intent anchoring, and scene-structure anchoring. During training, the planner receives current surround-view RGB images, historical trajectories, and navigation commands, and predicts future trajectories as discrete coordinate tokens. Two auxiliary branches use ground-truth future trajectories and frozen DINOv3 features, respectively, to supervise the same vision encoder.

The diagram distinguishes training supervision from forward prediction. The two anchors are parallel auxiliary constraints, not sequential inference tasks that first predict intent and then reconstruct roads. After joint training, neither ground-truth future trajectories nor the DINOv3 expert branch is needed to produce a plan.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Diverse data construction"] --> B["Surround-view images<br/>Vision encoder"]
    A -->|Ground-truth future trajectory| C["Driving-intent anchoring"]
    B --> C
    B --> D["Scene-structure anchoring"]
    A -->|Images for frozen DINOv3| D
    C -->|Training constraint| E["Joint training"]
    D -->|Training constraint| E
    B --> F["VLM trajectory prediction<br/>History and navigation commands"]
    F -->|Trajectory supervision| E
    F --> G["Future trajectory"]

Key Designs¶

1. Diverse data construction: combine atmospheric coverage, trajectory supervision, and scene semantics in one source

DiverseAD contains 150K driving scenes, approximately 1M videos, and 150M frames, spanning 9 atmospheric categories, 14 road types, and speeds of 0–180 km/h. Atmospheric categories extend beyond rain, snow, and fog to include normal sunny conditions, darkness, bright light, glare, dust, and wind. The authors distinguish bright from glare: the former mainly denotes intense daytime illumination, whereas the latter refers to strong lights in low-light or nighttime settings. This captures exposure and lighting variation rather than weather labels alone.

The collection vehicle carries 7 cameras, LiDAR, IMU, and GPS. Its forward cameras have fields of view of 30 degrees and 120 degrees; the other 5 cameras cover the front sides, rear sides, and rear. Processing establishes timestamp, pose, and environmental metadata before spatiotemporal alignment and sensor quality checks. Sparse or severely distorted point clouds, drifting GPS signals, and noisy IMU sequences are filtered, while visual-feature and trajectory similarity identify duplicate segments. Recording hardware should not be confused with planner input: the reported experiments use 6 image views, selecting the 30-degree forward camera rather than feeding all 7 cameras to the model.

Perception annotations use synchronized point clouds and images to label 3D boxes, lane lines, and road boundaries, followed by expert review and algorithmic checks. VQA construction organizes 3D objects and attributes into a scene graph, computes spatial relationships through BEV projection, and uses manually designed existence, counting, and status templates. Depth-first search instantiates questions and answers; VLM analysis and manual inspection then filter logical errors and improve phrasing. The answers thus have geometric grounding instead of being freely invented by a language model from images. Nevertheless, template-based question coverage is not equivalent to unrestricted driving reasoning.

2. Driving-intent anchoring: associate similar trajectory-direction patterns with similar visual features

Comparing absolute trajectory coordinates can mistake a positional offset for a difference in intent. The method first computes displacements between consecutive waypoints, normalizes each displacement to a unit direction, and measures intent similarity through the average cosine similarity of corresponding directions. For samples with \(K\) waypoints, it defines:

\[ S_{\mathrm{int}}^{ij}=\frac{1}{K-1}\sum_{k=1}^{K-1}\hat{v}_k^i\cdot\hat{v}_k^j, \qquad w_{\mathrm{int}}^{ij}=\frac{S_{\mathrm{int}}^{ij}+1}{2}. \]

Here, \(\hat{v}_k^i\) denotes, for sample \(i\), the unit displacement direction of segment \(k\); \(S_{\mathrm{int}}^{ij}\) lies in \([-1,1]\), and the mapped weight lies in \([0,1]\). Two samples can therefore receive a high association weight without sharing a location, provided their directional evolution is similar. Normalized vision-encoder features also produce pairwise cosine similarities within the batch, followed by a temperature-scaled softmax distribution. The authors describe the constraint as a soft-weighted KL term: trajectory-derived weights guide this distribution instead of dividing samples into hard positive and negative pairs.

The intuition is to preserve visual information consistent with turning patterns without relying excessively on the weather at capture time. Unit directions discard displacement magnitude, however, so this definition does not fully express acceleration or deceleration intent. The main text also does not specify how zero displacement during a stop is normalized. This is a direction-based supervisory proxy, not a complete and unambiguous driving-intent label.

3. Scene-structure anchoring: supplement road-layout constraints through a frozen expert's sample relationships

Similar directions do not imply similar road scenes, so intent anchoring alone may pull structurally different samples too close together. The second branch feeds the same image batch to frozen DINOv3, extracts and normalizes expert features, computes pairwise inner products as structure similarities, and converts them to soft weights through a Gaussian kernel. The primary encoder still learns its own batchwise feature-similarity distribution, now guided by the expert. This distills relationships between samples rather than requiring elementwise copying of every DINOv3 token or training an additional road-reconstruction network.

The anchors are complementary: trajectory relationships emphasize action direction, while expert relationships supply scene-layout information. DINOv3 serves as an empirically stable reference, not a proven geometric invariant. The local full-text extraction damages the Gaussian-kernel and both KL equations. This note therefore preserves the verifiable computational sequence without inventing the kernel's distance term or weight-normalization rule; exact reproduction requires checking the original PDF and implementation.

Loss & Training¶

Planning is formulated as question-answer-style sequence prediction. The BEV plane is quantized into output waypoint-coordinate tokens, supervised by cross-entropy against the ground-truth future trajectory, while the vision features receive the two auxiliary constraints. Following the textual description of the losses and their weights, the overall objective can be written as:

\[ \mathcal{L}=\mathcal{L}_{\mathrm{traj}}+\lambda_{\mathrm{int}}\mathcal{L}_{\mathrm{int}}+\lambda_{\mathrm{str}}\mathcal{L}_{\mathrm{str}}. \]

The coefficients control auxiliary-constraint strength, temperature controls the sharpness of the visual-similarity distribution, and kernel bandwidth controls how structure weights vary. The main text does not provide numerical settings for these hyperparameters, coordinate-grid resolution, or the complete training configuration, so this description is not a directly reproducible implementation specification. Its explicit benefit is to confine the extra computation to training: inference still uses the original planner to output future waypoints from current observations, trajectory history, and commands.

Key Experimental Results¶

Main Results¶

DiverseAD uses an 8:2 training/test split. Inputs are the past 2 seconds of trajectory and 6 current camera views, with prediction over the next 3 seconds. L2 measures trajectory-position error in m, lower being better; Avg. is the reported average over the 1-, 2-, and 3-second results. VQA Acc. is question-answer accuracy, higher being better. Baselines marked * have all been fine-tuned on DiverseAD, so this table does not contrast unadapted models against the proposed model.

DiverseAD method	1 s L2 ↓	2 s L2 ↓	3 s L2 ↓	Average L2 ↓	VQA Acc. (%) ↑
UniAD*	1.04	2.06	3.23	2.11	Unsupported
VAD*	0.72	1.67	2.69	1.69	Unsupported
OpenEMMA*	0.71	1.60	2.67	1.66	60.28
OmniDrive*	0.61	1.45	2.50	1.52	64.51
FSDrive*	0.59	1.38	2.34	1.44	65.33
Ours	0.53	1.21	1.99	1.24	72.68

Relative to FSDrive*, average L2 decreases by 0.20 m, approximately a 13.9% relative reduction; 3-second L2 decreases by 0.35 m. VQA accuracy increases by 7.35 percentage points, not a 7.35% relative gain. In the atmospheric breakdown, average L2 under darkness is 1.30 m versus 1.60 m, and under glare it is 1.51 m versus 1.66 m, showing that the aggregate gain does not come solely from normal weather.

nuScenes uses the ST-P3 protocol for L2 and collision-rate computation, with no additional ego state beyond trajectory history. The paper reports average L2 of 0.39 m for its method versus 0.40 m for OmniDrive, and an average collision rate of 0.16%, tied with World4Drive at 0.16%. The proposed method is not best at every horizon: its 3-second collision rate is 0.41%, above World4Drive's 0.33%. The two datasets differ in difficulty and statistical distribution, so 0.39 m versus 1.24 m should not be interpreted directly as a difference in the method's underlying capability.

Ablation Study¶

The table selects component and alternative-design analyses on DiverseAD. All results use average L2 in m, lower being better; increases are calculated by subtracting the full model's 1.24 m from each reported result.

Config	Average L2 ↓	Increase over full model	Note
Full model	1.24	0.00	Both soft-weighted anchors
Without driving-intent anchoring	1.39	0.15	No intent constraint
ADE-based intent similarity	1.38	0.14	Average displacement error replaces direction relationships
Without scene-structure anchoring	1.42	0.18	No structure constraint
Gram-matrix structure similarity	1.35	0.11	Alternative expert-relationship computation
Hard contrastive intent constraint	1.38	0.14	Pairs constructed with cosine threshold 0.5
Hard contrastive structure constraint	1.33	0.09	Pairs constructed with cosine threshold 0.5

Key Findings¶

Removing structure anchoring adds 0.18 m, while removing intent anchoring adds 0.15 m. Both references contribute, but the difference does not establish that one is more important in every scenario.
ADE and hard-contrastive replacements both underperform the full design. This supports direction relationships with continuous soft weights rather than merely adding an arbitrary auxiliary loss.
Initial baselines incur large errors on DiverseAD and improve substantially after fine-tuning. Data adaptation therefore matters, and the entire improvement from original checkpoints to the proposed method cannot be attributed to the dual-anchor mechanism.

Highlights & Insights¶

The dataset combines planning trajectories with atmospheric variation, avoiding the limitation of weather-rich data annotated only for perception. Geometrically grounded questions further connect scene understanding and trajectory planning within the same source.
Auxiliary supervision focuses on sample relationships rather than requiring a hand-designed network branch for every atmospheric category. Soft weights preserve continuous similarity in driving actions and layouts instead of forcing hard positive/negative decisions.
The expert and future trajectories serve only as training supervision, not inference inputs. This provides representation regularization without establishing that the complete planner operates in real time.

Limitations & Future Work¶

The local cache includes the complete main paper and references but not the repeatedly cited supplementary material or implementation details; equation extraction is also damaged. Exact losses, feature-token aggregation, hyperparameters, and VQA scoring rules still require verification against the original implementation or supplement.
Reported evidence primarily concerns open-loop trajectory error and offline collision rates, not closed-loop driving safety. The main text does not clearly demonstrate leave-one-weather-out, cross-city, or route-isolated testing, so cross-weather generalization should not be read as proven generalization to unseen atmospheric categories.
Weather is not always irrelevant to actions: rain and snow affect braking and traction, while direction normalization discards speed magnitude. The anchors are useful but incomplete proxies whose scope needs validation under actual driving conditions.
Dataset access requires an application form and contact with the authors, rather than an unrestricted full-download package supplied in the paper. Further disclosure of access conditions, licensing, and training configuration will directly affect external reproduction and comparison.

vs nuScenes / adverse-weather perception datasets: this work combines broader environmental coverage with ego trajectories, perception, and VQA annotations. Its advantage is more complete task connectivity, but scale alone cannot replace rigorous out-of-distribution testing.
vs OmniDrive / FSDrive: these share the vision-language-model route to scene understanding and planning, whereas this paper emphasizes atmospheric robustness in learned representations. Fair comparison should prioritize baselines fine-tuned on DiverseAD rather than only showing domain-shift failures of original checkpoints.
vs hard contrastive learning / featurewise distillation: intent and expert relationships provide continuous supervision instead of binary sample labels or pointwise feature alignment. Ablations support this relational constraint, while the precise KL and kernel implementation still needs alignment with the original materials.

Rating¶

Novelty: 4/5. The combination of large-scale multi-weather planning data and dual-anchor relational supervision is well targeted, though its underlying techniques largely build on established representation-learning mechanisms.
Experimental Thoroughness: 3/5. Cross-dataset results, atmospheric breakdowns, and several ablations are included, but closed-loop evaluation, isolated generalization tests, and reproduction details remain insufficient.
Writing Quality: 3/5. The data and method narrative is clear, while some equations and implementation information need fuller explanation.
Value: 4/5. The work is useful for weather-robust end-to-end driving research, with practical impact dependent on data access and reproducibility.