VVSim: A Large-Scale Aerial-Ground Dataset and Benchmark for Cooperative Perception¶

Conference: ECCV 2026
Paper: ECCV Original
Code: https://github.com/LOTEAT/vvdetection3d
Area: Autonomous Driving
Keywords: Aerial-Ground Cooperative Perception, Autonomous Driving, Multimodal Fusion, Cooperative 3D Detection, Benchmark Dataset

TL;DR¶

Addressing the view constraints and occlusion vulnerabilities of conventional ground-based cooperative perception, this paper presents VVSim, the first large-scale aerial-ground cooperative perception dataset comprising 61k time-synchronized multimodal frames, alongside VVFormer, a novel framework leveraging cross-view voxel projection, height-aware attention, and agent-aware deformable cross-attention to advance cooperative 3D perception.

Background & Motivation¶

Autonomous driving systems rely on precise and real-time environmental perception to execute safe navigation and control decisions. However, single-vehicle sensing is fundamentally constrained by sensor range and persistent line-of-sight occlusions, frequently generating hazardous blind spots in cluttered urban traffic. While vehicle-to-vehicle (V2V) cooperation enables connected autonomous vehicles (CAVs) to exchange sensory features to expand coverage, its performance heavily depends on the spatial density and positioning of participating CAVs. Vehicle-to-infrastructure (V2I) approaches overcome some of these constraints by deploying roadside units (RSUs), but the fixed mounting viewpoints of RSUs lack adaptability in dynamic environments and impose prohibitive deployment and maintenance costs.

Unmanned aerial vehicles (UAVs) provide a transformative paradigm by offering agile maneuverability, variable flight altitudes, and flexible top-down perspectives. Elevated aerial viewpoints naturally penetrate dense ground occlusions and capture panoramic spatial context inaccessible to ground sensors. Nevertheless, the development of aerial-ground cooperative perception (AGCP) has been critically hindered by two main bottlenecks: the absence of large-scale, high-fidelity multimodal benchmarks capturing diverse environments and safety-critical corner cases, and the lack of cooperative perception architectures capable of handling heterogeneous geometric transformations between aerial downward-facing monocular cameras and ground-level LiDAR/camera rigs.

To overcome these challenges, this paper develops both a comprehensive benchmark dataset and a dedicated cooperative perception architecture. Built on an extended AirSim platform with realistic multi-agent coordination, VVSim provides rich multimodal data spanning multiple UAV flight regimes. In parallel, the authors introduce VVFormer, a tailored framework for aerial-ground collaboration. The core idea is to project high-altitude aerial imagery into ground BEV coordinates via explicit 3D voxel geometric alignment and height-aware self-attention, and dynamically select the most informative multi-agent features using agent-aware deformable cross-attention to achieve seamless aerial-ground complementary perception.

Method¶

Overall Architecture¶

VVFormer unifies multi-agent multimodal observations from both aerial UAVs and ground vehicles into a shared bird's-eye-view (BEV) representation for 3D scene understanding. The pipeline consists of four sequential stages: metadata sharing and synchronization, heterogeneous feature extraction, cross-view attention with height-aware aggregation (CVA), and agent-aware deformable cross-attention (AAA). Initially, all participating agents broadcast their global poses, timestamps, and sensor calibration matrices. Ground vehicles encode LiDAR point clouds into BEV feature representations using PointPillars, while UAVs extract dense visual representations from downward-facing RGB images using a Swin Transformer. The Cross-View Attention (CVA) module projects image features into a 3D voxel grid and compresses them vertically into a unified ground-aligned BEV map using height-wise self-attention. Finally, the Agent-Aware Attention (AAA) module estimates spatial importance scores across agents, adaptively aggregates the most informative features, and refines them via spatial cross-attention before feeding them into a 3D detection head.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input: UAV Downward Images + CAV LiDAR Sweeps"] --> Pre["Feature Extraction: Swin Transformer & PointPillars BEV Encoders"]
    Pre --> CVA["Cross-View Feature Alignment<br/>3D Voxel Projection & Bilinear Feature Sampling"]
    CVA --> HAF["Height-Aware Self-Attention Fusion<br/>Positional Encoding & Vertical Self-Attention Compression"]
    HAF --> AAA["Agent-Aware Attention Scoring<br/>Deformable Cross-Attention & ScoringNet Selection"]
    AAA --> SCA["Spatial Cross-Attention Refinement<br/>Reference Point Deformable Sampling & BEV Context Aggregation"]
    SCA --> Out["3D Detection Head Output: Class / 3D Bounding Boxes / Velocities"]

Key Designs¶

1. Cross-View Voxel Projection and Feature Alignment: Unifying Aerial-Ground Geometry The substantial difference in vantage point and pitch angle between high-altitude UAVs and ground vehicles invalidates conventional planar homography or ground-plane assumptions. To establish rigorous spatial correspondence, the model extends the ground LiDAR BEV pillar grid along the vertical axis, creating a 3D voxel grid of size $H_l \times W_l \times D_{vox}$. For each voxel center $p_v = (x_v, y_v, z_v, 1)^\top$ in global coordinates, the UAV's extrinsic matrix $T_{ext} \in \mathbb{R}^{4 \times 4}$ transforms it into the UAV camera coordinate frame $p_c = T_{ext} p_v = (x_c, y_c, z_c, 1)^\top$. Next, the intrinsic matrix $T_{int} \in \mathbb{R}^{3 \times 3}$ projects the 3D point onto homogeneous image coordinates $[u_h, v_h, w_h]^\top = T_{int} [x_c, y_c, z_c]^\top$, which are normalized into pixel coordinates: $$u = \frac{u_h}{w_h}, \quad v = \frac{v_h}{w_h}$$ A validity mask $\mathcal{M}(x_v, y_v, z_v)$ filters out voxels outside the image bounds or behind the image plane. For all valid voxels, bilinear interpolation samples features from the backbone-downsampled UAV feature map $F_i^c$, yielding a 3D voxel-aligned tensor $F_i^{c2v} \in \mathbb{R}^{D_{vox} \times H_l \times W_l \times C}$ geometrically anchored to the ground LiDAR grid.

2. Height-Aware Self-Attention Fusion: Vertical Context Aggregation Collapsing vertical voxels via simple average pooling discards height-dependent structural cues, creating depth ambiguity beneath overpasses, tree canopies, or tall building facades. To retain vertical geometry, the model injects a sinusoidal 1D height positional encoding $P_{height}(d) \in \mathbb{R}^C$ into each height slice of the voxel feature volume: $$\tilde{F}_i^{c2v}(d, h, w) = F_i^{c2v}(d, h, w) + P_{height}(d)$$ The encoding frequencies vary geometrically across feature channels. Next, for every BEV cell $(h, w)$, self-attention is applied along the height dimension $D_{vox}$. Linear projections map $\tilde{F}_i^{c2v}$ into query $Q$ and key $K$ vectors, deriving normalized attention weights: $$A(d, d') = \frac{\exp((Q_{h,w,d} \cdot K_{h,w,d'}) / \sqrt{C})}{\sum_{d''=1}^{D_{vox}} \exp((Q_{h,w,d} \cdot K_{h,w,d''}) / \sqrt{C})}$$ The height-aggregated BEV feature is computed via weighted summation $F_i^{fused}(h, w) = \sum_{d=1}^{D_{vox}} A(d, :) V_{h,w,d}$, compressing 3D vertical information into a rich 2D BEV feature map $F_i^{fused} \in \mathbb{R}^{C \times H_l \times W_l}$.

3. Agent-Aware Deformable Cross-Attention: Dynamic Multi-Agent Contribution Scoring In heterogeneous collaborative networks, individual agents offer disparate observation quality across different regions of the BEV grid—an overhead UAV provides broad contextual visibility with coarser resolution, whereas a nearby vehicle yields dense point clouds but suffers severe distant occlusions. To dynamically aggregate the most informative observations, the Agent-Aware Attention (AAA) module first computes deformable cross-attention (DCA) between the flattened ego query $X_{ego} \in \mathbb{R}^{S \times C}$ ($S = H_l \times W_l$) and collaborator $j$'s feature $X_j$: $$X_j^{out}(q) = \sum_{m=1}^M W_m \sum_{k=1}^K A_{mqk} X_j(p_q + \Delta p_{mqk})$$ where $q$ denotes the query grid index, $M$ and $K$ denote the number of attention heads and sampling points per head, and $\Delta p_{mqk}$ and $A_{mqk}$ are learnable sampling offsets and normalized attention weights. The output $X_j^{out}$ is passed to a lightweight ScoringNet to predict an importance score map $S_j \in \mathbb{R}^{S \times 1}$. The system dynamically selects the highest-scoring agent at each grid position: $$X^*(q) = X_{j^*(q)}^{out}(q), \quad \text{where } j^*(q) = \arg\max_j S_j(q)$$ Subsequently, a spatial cross-attention layer refines $X^*(q)$ around normalized reference points using learnable 2D offsets, ensuring spatial smoothness and boundary continuity across collaborating perspectives.

Loss & Training¶

The framework is trained end-to-end with a perception range set to $x, y \in [-50, 50]$ meters and a BEV grid resolution of 0.5 meters. The model is optimized for 24 epochs using AdamW with an initial learning rate of $0.001$ and a weight decay of $0.01$. The multi-task objective combines smooth L1 loss for 3D bounding box regression, focal loss for object category classification, and an L1 regression loss for velocity estimation.

Key Experimental Results¶

Main Results¶

Evaluation was conducted on the VVSim benchmark test set (18,500 frames of complex multi-agent traffic sequences), following the standard nuScenes evaluation protocol: mean Average Precision (mAP), mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE), and nuScenes Detection Score (NDS).

Method	Publication	mAP ↑	NDS ↑	mATE ↓	mASE ↓	mAOE ↓	mAVE ↓
F-Cooper	SEC 2019	0.5865	0.4773	0.3530	0.1144	0.6915	2.7622
V2X-ViT	ECCV 2022	0.6150	0.4847	0.3455	0.1061	0.7766	2.6077
CoBEVT	CoRL 2023	0.5777	0.4654	0.3681	0.1544	0.7124	2.9697
SICP	IROS 2024	0.6327	0.5144	0.2263	0.1055	0.6877	3.5578
L4DR	AAAI 2025	0.6420	0.4924	0.3595	0.1408	0.7862	2.9374
VVFormer (Ours)	ECCV 2026	0.6490	0.5268	0.2168	0.0954	0.6647	3.2290

Ablation Study¶

Ablation experiments on the VVSim test set isolate the individual contributions of UAV observations (UAVs), Cross-View Attention (CVA), and Agent-Aware Attention (AAA):

Config	CVA	AAA	UAVs	mAP ↑	NDS ↑	mATE ↓	mASE ↓	mAOE ↓	mAVE ↓
Full model	✓	✓	✓	0.6490	0.5268	0.2168	0.0954	0.6647	3.2290
w/o UAVs	✓	✓	✗	0.5912	0.4912	0.2459	0.0991	0.6992	3.3559
w/o AAA	✓	✗	✓	0.6063	0.4964	0.2613	0.1089	0.6972	2.5410
w/o CVA	✗	✓	✓	0.5294	0.4550	0.2964	0.1286	0.6720	3.2534

Key Findings¶

Crucial Role of Aerial Perspectives: Omitting UAV data (w/o UAVs) causes mAP to fall by 5.78% (from 0.6490 to 0.5912) and NDS to decrease by 3.56% (from 0.5268 to 0.4912), underscoring that aerial top-down observations successfully mitigate severe ground occlusions and expand global spatial coverage.
Primacy of Geometric Alignment: Removing CVA induces the steepest degradation across all metrics, with mAP collapsing by 11.96% (down to 0.5294) and NDS dropping by 7.18% (down to 0.4550). This demonstrates that without explicit 3D voxel projection and vertical height-aware attention, unaligned aerial features inject severe spatial noise into ground BEV grids.
Superiority of Dynamic Agent Selection: Disabling the AAA module leads to a 4.27% drop in mAP (to 0.6063), validating that ScoringNet-driven dynamic selection effectively discards low-quality or redundant features from oblique or distant vantage points.

Highlights & Insights¶

First Comprehensive AGCP Benchmark: VVSim supplies 61k fully annotated frames, 255k LiDAR sweeps, and 3.5M multi-modal images across 11 diverse scenes and 5 weather conditions with varied UAV operational regimes (hovering, cyclic, escort), filling a long-standing void in collaborative autonomous driving datasets.
Robust Height-Aware Projection: Combining 3D voxel projection with 1D vertical sinusoidal encoding and height-axis self-attention effectively resolves vertical parallax ambiguities between aerial imagery and ground coordinates.
Generalizable Scoring Paradigm: The AAA module's design—coupling deformable cross-attention with localized feature scoring—provides an efficient, transferable pattern for heterogeneous multi-agent sensor fusion across mixed robotics domains.

Limitations & Future Work¶

Admitted Limitations: Dynamic vehicles across VVSim exhibit high velocity variance ranging from slow stop-and-go maneuvers to high-speed transit. Consequently, the velocity estimation error (mAVE of 3.2290) remains slightly inferior to specialized temporal baselines.
Real-World Sim-to-Real Gap: VVSim is generated within an augmented AirSim environment. Physical real-world factors such as wind turbulence, severe camera vibration, GPS/IMU pose drift, and communication transmission latency require further exploration.
Future Directions: Incorporating multi-frame temporal feature queues could significantly refine velocity estimation, while communication-efficient feature pruning could minimize transmission bandwidth in real-world aerial-ground links.

vs OPV2V / V2X-ViT / CoBEVT: Existing collaborative perception benchmarks focus predominantly on symmetric V2V or static V2I topologies; VVFormer successfully establishes cross-domain fusion between highly asymmetric aerial and ground platforms.
vs CoPeD / V2U-COO / Griffin: Earlier aerial-ground datasets were restricted to very low UAV altitudes (2–10 m), monocular setups, or simple interaction trajectories; VVSim features high-altitude multi-drone flights across 19 critical interaction scenarios.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ (Pioneering large-scale aerial-ground cooperative perception benchmark with height-aware cross-view transformer design)
Experimental Thoroughness: ⭐⭐⭐⭐☆ (Comprehensive validation on 18.5k test frames with full nuScenes metrics and multi-component ablations)
Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous mathematical formulation, clear architectural narrative, and thorough experimental breakdown)
Value: ⭐⭐⭐⭐⭐ (Essential foundational benchmark and codebase for future vehicle-to-everything and low-altitude collaborative mobility research)