Skip to content

title: >- [Paper Note] Multi-view Multi-vehicle Driving Dataset for Novel View Synthesis description: >- [ECCV 2026][3D Vision][Novel View Synthesis] Introduces MV2, a multi-vehicle multi-platform driving dataset capturing synchronized views from a car, scooter, and drone to establish a rigorous extrapolation benchmark for driving NVS and camera pose estimation. tags: - ECCV 2026 - 3D Vision - Autonomous Driving - Novel View Synthesis - 3D Gaussian Splatting - Dataset & Benchmark date: 2026-09-19 content_hash: e00b9a0761777a7b

Multi-view Multi-vehicle Driving Dataset for Novel View Synthesis

Conference: ECCV 2026
Paper: CVF Open Access
Code: https://mv2-dataset.github.io/
Area: 3D Vision
Keywords: Autonomous Driving NVS, Multi-Vehicle Multi-View, Cross-Trajectory Extrapolation, 3D Gaussian Splatting, Pose Estimation Benchmark

TL;DR

Addressing the fundamental limitation that existing driving NVS benchmarks evaluate only temporal interpolation along a single vehicle trajectory, this paper presents MV2, the first real-world multi-vehicle driving dataset captured synchronously from a four-wheeler, a two-wheeler, and an aerial drone, introducing a rigorous cross-trajectory and aerial-to-ground extrapolation benchmark where state-of-the-art models experience dramatic performance degradation under wider baselines.

Background & Motivation

Novel view synthesis (NVS) powered by differentiable rendering frameworks such as NeRF and 3D Gaussian Splatting (3DGS) has shown transformative potential for autonomous driving, serving as the core engine for photorealistic closed-loop simulators, corner-case stress testing, and sensor re-simulation. However, training and evaluating NVS in unbounded outdoor driving scenarios is notoriously difficult due to forward-moving monocular camera configurations, pervasive dynamic traffic agents, and vast untextured regions such as the sky. More critically, existing driving NVS benchmarks—typically repurposed from standard datasets such as KITTI, Waymo Open Dataset, nuScenes, or Argoverse 2—are universally constrained to a single ego-vehicle trajectory where test frames are selected via temporal subsampling (e.g., holding out every \(k\)-th frame). Consequently, these setups only measure trajectory interpolation, concealing whether models can extrapolate to genuinely unseen lateral lanes or distinct elevation viewpoints.

Prior attempts to circumvent this single-trajectory limitation suffer from major compromises: synthetic datasets (such as CARLA-based XLD) lack real-world optical artifacts and complex multi-path lighting; multi-traversal protocols (such as MTGS and Para-lane) revisit the same road at different timestamps, which inevitably breaks temporal simultaneity and forces the manual masking of all dynamic vehicles and pedestrians; and recording an identical dynamic traffic scene across multiple lateral viewpoints using a single car is physically impossible. Furthermore, existing aerial-to-ground datasets either lack metric camera poses or are restricted to static landmark reconstruction. The community has thus lacked a multi-perspective physical benchmark that simultaneously captures real dynamic traffic, wide physical baselines, and certified sub-pixel geometric registration.

To resolve these challenges, this paper introduces a coordinated multi-vehicle convoy capture strategy incorporating a four-wheel passenger car, a two-wheel scooter, and an aerial drone. The car and scooter travel in separate adjacent lanes to introduce substantial physical lateral disparity, while the drone flies overhead to provide occlusion-free holistic context. By coupling this setup with a robust two-step Structure-from-Motion (SfM) localization framework and strict epipolar consistency verification, the core idea is to establish MV2—a high-fidelity multi-view multi-vehicle dataset spanning 50 dynamic urban sequences and 12,000 images—defining rigorous cross-vehicle and aerial-to-ground evaluation protocols that expose the severe extrapolation bottlenecks of both optimization-based and feed-forward NVS and pose estimation frameworks.

Method

Overall Architecture

The MV2 data generation and benchmarking pipeline encompasses multi-sensor time-synchronized acquisition, decoupled two-stage coordinate frame registration, epipolar error-bounded geometric verification, and bidirectional cross-platform extrapolation evaluation. The recording platform deploys calibrated GoPro 10 cameras across three vehicles: a car with stereo forward cameras (left \(V^L\), central \(V^C\)), a scooter with a forward camera \(V^S\), and a drone with a forward-facing camera \(V^D\). High-definition 1080p footage is collected under wall-clock synchronization, segmented into 100-frame sequences, registered into unified scene-level coordinate frames, and pruned via epipolar error filtering. Downstream evaluation is established through two primary protocols: Eval-Car-Train (training on central car camera \(V^C\) and testing across lateral car \(V^L\), cross-lane scooter \(V^S\), and drone \(V^D\)) and Eval-Drone-Train (training on drone \(V^D\) and testing on ground vehicles \(V^C\) and \(V^S\)).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Vehicle Synchronized Capture<br/>Car(VC,VL) + Scooter(VS) + Drone(VD)"] --> B["Two-Stage Decoupled Registration<br/>Independent Train SfM + Test Localizer"]
    B --> C["Human-in-the-Loop Epipolar Verification<br/>Dense RoMA Matching with Error Thresholding"]
    C --> D["Cross-Platform Extrapolation Benchmark<br/>Eval-Car-Train and Eval-Drone-Train"]

Key Designs

1. Multi-Vehicle Synchronized Convoy Data Acquisition: Capturing True Cross-Trajectory Dynamics To observe identical moving vehicles and pedestrian flows from disparate viewpoints simultaneously, MV2 deploys a synchronized convoy comprising a passenger car, a two-wheel scooter, and an aerial drone across diverse urban and semi-urban routes over 5 days (yielding over two hours of driving footage). All cameras record at 1080p 60 FPS and are hard-synchronized to system wall-clock time. The car and scooter maintain adjacent lane spacing to generate realistic lateral baseline diversity, while the drone follows overhead along the corridor. The continuous streams are uniformly downsampled to 2 FPS and split into 100-frame snippets, excluding segments with severe traffic gridlock, underpass/tunnel signal losses, lateral convoy misalignments, and red-light halts. This yields 200 initial five-camera sequence candidates with rich co-visibility.

2. Two-Stage Decoupled Camera Pose Registration: Preventing Wide-Baseline Reconstruction Degeneracy Jointly registering all sensors simultaneously using global Structure-from-Motion (SfM) causes severe optimization failures because extreme baseline variations between ground and aerial views lead to dense feature mismatching and scale drift. MV2 decouples this into a two-stage registration pipeline: First, the training sequence for a given protocol (\(V^C\) in Eval-Car-Train or \(V^D\) in Eval-Drone-Train) is reconstructed independently using standard COLMAP SfM, producing an accurate, self-consistent 3D coordinate frame and training camera extrinsics. Second, test frames from alternate cameras (\(V^L, V^S, V^D\)) are localized into this frozen coordinate frame via COLMAP's visual localization module using their 15 nearest training frames as visual anchors. This decoupled design protects training trajectory integrity from cross-view optimization corruption while allowing selective rejection of mislocalized test frames.

3. Dense Neural Feature Matching and Epipolar Verification: Certifying Sub-Pixel Metric Fidelity In real-world driving environments without millimeter-accurate RTK-GPS or motion capture ground truth, naive relative pose metrics can be misleading if the reference anchor itself drifts. MV2 models the training trajectories as sequential pose graphs, validating consecutive frames and long-range loop closures (e.g., between the 1st and 50th/100th frames). Human annotators delineate corresponding 3D physical regions via bounding boxes across image pairs, inside which dense pixel correspondences \(X_{\text{RoMA}}\) are extracted using the foundation matcher RoMA. For any correspondence pair \((x_1, x_2)\), camera intrinsics \(K_1, K_2\), and relative extrinsics \((R, t)\), the fundamental matrix is defined as \(F = K_2^{-\top} [t]_\times R K_1^{-1}\), and the epipolar distance is: $\(e = \|x_2^\top F x_1\|\)$ The mean error \(e_a\) and maximum error \(e_m = \max_i e_i\) across all \(N\) pairs are recorded. Sequences are accepted only when every consecutive training pair satisfies a strict hard threshold of \(e_m \le 30\) pixels (retaining 50 pristine scenes out of 200 candidates, totaling 12,000 images). Test frames are subsequently verified against their nearest training frame under the same \(e_m \le 30\) filter, guaranteeing that rendering degradations reflect synthesis failures rather than pose artifacts.

4. LiDAR-Free Depth Adaptation and Motion Mask Disentanglement: Enabling Monocular Driving 3DGS Because MV2 operates without bulky LiDAR scanners, the pipeline adapts cutting-edge vision priors to supervise optimization-based driving 3DGS models such as PVG. Monocular depth predictions from Depth Anything V3 (DaV3) are scaled and aligned to sparse COLMAP SfM depth using a per-scene affine scale-and-shift objective: $\(\mathcal{L}_{\text{depth}} = \sum_{p \in \mathcal{P}} \|\omega D_{\text{DaV3}}(p) + \varepsilon - D_{\text{SfM}}(p)\|_1\)$ where \(\mathcal{P}\) represents valid sparse SfM points, and \(\omega, \varepsilon\) are scene-specific parameters. A calibrated distance filter restricting valid depth to \([10, 30]\text{m}\) eliminates scale ambiguity from infinite sky and near-field motion blur. Dynamic moving objects are segmented by intersecting DroneSplat multi-view motion masks with Segment Anything Model (SAM) prompt masks, producing high-quality dynamic supervision (achieving 63.69% IoU against ground-truth dynamic masks) that allows dynamic models to isolate moving agents without 3D bounding boxes.

Key Experimental Results

Main Results

Eight representative NVS methods—spanning feed-forward generalizable 3DGS (DepthSplat, MonoSplat, MVSplat), classic volume rendering (NeRF), and optimization-based 3DGS (splatfacto, DesireGS, PVG, droneSplat)—are evaluated across the Eval-Car-Train and Eval-Drone-Train benchmarks. Test splits are categorized into standard trajectory interpolation (\(T^{C \to C}, T^{D \to D}\)), small-baseline car-chassis shift (\(T^{C \to L}\)), cross-vehicle lateral shift (\(T^{C \to S}, T^{D \to S}\)), and aerial-to-ground shift (\(T^{D \to C}\)).

Table 1: Quantitative NVS benchmark comparison across test splits (corresponding to Table 1 in the original paper)

Protocol Model PSNR \(\uparrow\) (\(T^{\to \text{Self}}\)) PSNR \(\uparrow\) (\(T^{\to \text{Lateral}}\)) PSNR \(\uparrow\) (\(T^{\to \text{Cross-Vehicle}}\)) SSIM \(\uparrow\) (\(T^{\to \text{Cross}}\)) LPIPS \(\downarrow\) (\(T^{\to \text{Cross}}\))
Eval-Car-Train DepthSplat (12v) 18.05 17.13 14.39 0.25 0.40
MonoSplat (12v) 18.43 18.24 15.71 0.28 0.33
MVSplat (12v) 16.33 14.08 12.88 0.35 0.54
NeRF (nerfacto) 20.93 19.38 15.73 0.52 0.59
3DGS (splatfacto) 24.22 22.52 16.78 0.64 0.47
DesireGS 25.96 24.31 19.48 0.55 0.34
PVG (SOTA) 27.01 25.44 20.23 0.58 0.31
Eval-Drone-Train DepthSplat (12v) 16.78 9.09 (\(T^{D \to C}\)) 7.87 (\(T^{D \to S}\)) 0.14 0.56
MonoSplat (12v) 18.54 10.21 (\(T^{D \to C}\)) 8.32 (\(T^{D \to S}\)) 0.23 0.52
3DGS 29.13 10.36 (\(T^{D \to C}\)) 11.42 (\(T^{D \to S}\)) 0.43 0.46
NeRF 21.30 11.01 (\(T^{D \to C}\)) 11.54 (\(T^{D \to S}\)) 0.35 0.61
droneSplat 16.43 12.16 (\(T^{D \to C}\)) 12.33 (\(T^{D \to S}\)) 0.50 0.72
PVG (SOTA) 28.12 12.65 (\(T^{D \to C}\)) 13.82 (\(T^{D \to S}\)) 0.46 0.52

Ablation Study

Table 2: Depth supervision source and filtering ablations for PVG on WOD and MV2 (corresponding to Table 2 in the original paper)

Dataset Split Depth Supervision Setting PSNR \(\uparrow\) SSIM \(\uparrow\) LPIPS \(\downarrow\)
Waymo (WOD) \(T^{C \to C}\) Multi-view LiDAR 28.45 0.912 0.145
Waymo (WOD) \(T^{C \to C}\) Single-view LiDAR 28.66 0.792 0.121
Waymo (WOD) \(T^{C \to C}\) DaV3 Depth + \([10, 30]\text{m}\) Filter 29.12 0.815 0.108
MV2 (Ours) \(T^{C \to C}\) COLMAP Sparse Depth 24.72 0.655 0.238
MV2 (Ours) \(T^{C \to C}\) DaV3 Depth + IQR Filter 25.88 0.680 0.210
MV2 (Ours) \(T^{C \to C}\) DaV3 Depth + \([10, 30]\text{m}\) Filter 27.01 0.720 0.200
MV2 (Ours) \(T^{C \to S}\) (Cross-Vehicle) DaV3 Depth + IQR Filter 19.43 0.550 0.330
MV2 (Ours) \(T^{C \to S}\) (Cross-Vehicle) DaV3 Depth + \([10, 30]\text{m}\) Filter 20.23 0.580 0.310

Table 3: Dynamic object mask supervision ablation on cross-vehicle test split \(T^{C \to S}\) (corresponding to Table 3 in the original paper)

Mask Supervision Dynamic Region PSNR \(\uparrow\) Dynamic Region SSIM \(\uparrow\) Full Image PSNR \(\uparrow\) Full Image SSIM \(\uparrow\) Full Image LPIPS \(\downarrow\)
None (No Mask) 13.98 0.32 14.72 0.34 0.63
DS+SAM (Pseudo Mask) 19.93 0.52 21.24 0.58 0.30
GT (Human Annotated Upper Bound) 21.70 0.62 24.29 0.72 0.18

Key Findings

  • Severe Performance Drop under Cross-Vehicle Extrapolation: Under Eval-Car-Train, the top-performing model PVG achieves 27.01 dB on same-trajectory interpolation (\(T^{C \to C}\)), drops slightly to 25.44 dB on small lateral vehicle displacement (\(T^{C \to L}\)), and plummets to 20.23 dB on the cross-lane scooter view (\(T^{C \to S}\))—a drop of nearly 7 dB. In the aerial-to-ground setup, performance drops precipitously from 28.12 dB (\(T^{D \to D}\)) to 12.65 dB (\(T^{D \to C}\)), underscoring that current driving NVS models rely heavily on viewpoint interpolation.
  • Optimization-Based Methods Decisively Outperform Feed-Forward 3DGS: Even when fed 12 context views, feed-forward generalizable models (MonoSplat, DepthSplat) hover between 14–15 dB on \(T^{C \to S}\), struggling with the geometric scale variations inherent in outdoor driving environments compared to scene-optimized representations.
  • Monocular Depth Priors Successfully Replace LiDAR: PVG trained with DaV3 monocular depth and a \([10, 30]\text{m}\) range filter reaches 27.01 dB on MV2 \(T^{C \to C}\), approaching Waymo LiDAR-supervised performance (28.66 dB). Combining DS+SAM pseudo masks boosts dynamic region PSNR from 13.98 dB to 19.93 dB, offering a scalable LiDAR-free training paradigm.
  • Feed-Forward Pose Estimators Break Down under Wide Baselines: Evaluating test pose localization reveals that while optimization-based COLMAP maintains an average epipolar error under 30 pixels, recent feed-forward estimators (MASt3r-sfm, MapAnything, VGGT) yield large errors exceeding 50 pixels, establishing MV2 as a demanding benchmark for camera pose estimation.

Highlights & Insights

  • First Physical Multi-Vehicle Synchronized Convoy Benchmark: By coordinating a car, an electric scooter, and an aerial drone, the dataset captures dynamic outdoor traffic simultaneously from diverse lanes and altitudes, breaking the structural limitation of single-trajectory driving datasets.
  • Two-Stage Decoupled SfM and Epipolar Verification: The registration pipeline decouples training graph reconstruction from test frame visual localization, validated by dense RoMA feature correspondences and an epipolar error bound (\(e_m \le 30\)), guaranteeing certified sub-pixel pose accuracy without motion capture rigs.
  • Exposing the Simulation Bottleneck for Lane-Change Scenarios: The quantitative results prove that even basic maneuvers such as lane changes generate degraded, blurry renderings in SOTA neural simulators, providing an empirical wake-up call for neural rendering in autonomous driving.

Limitations & Future Work

  • Constrained Environmental and Lighting Diversity: Due to safety requirements for multi-vehicle convoy synchronization, captures are restricted to daytime dry weather, leaving night, adverse rain, snow, and lens glare unaddressed.
  • Extreme Perspective Disparity in Aerial-to-Ground Synthesis: Severe vertical occlusions under dense canopies and high-rise street canyons degrade aerial-to-ground synthesis below 14 dB PSNR across all baselines, highlighting the need for generative priors or 3D diffusion outpainting.
  • Scale and Generalization Bottlenecks in Feed-Forward 3DGS: Feed-forward Gaussian models trained on DL3DV fail to scale to unbounded outdoor driving scenes, pointing toward future research in foundation geometric transformers integrated with scalable 3DGS representations.
  • vs Waymo / nuScenes / KITTI (Single-Trajectory Datasets): Conventional datasets record from a single vehicle chassis and evaluate via temporal subsampling on the same trajectory (interpolation); MV2 provides independent convoy trajectories that benchmark true spatial extrapolation across traffic lanes and elevations.
  • vs XLD / CARLA (Synthetic Benchmarks): XLD relies on synthetic CARLA rendering with simplified textures and idealized lighting; MV2 records physical roads with real camera sensors, motion blur, lens flare, and complex dynamic traffic.
  • vs MTGS / Para-lane (Multi-Traversal Static Datasets): Multi-traversal setups drive the same road repeatedly at different times and must erase dynamic objects; MV2 preserves synchronized dynamic traffic flow, enabling cross-view evaluation of moving vehicles and pedestrians.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering multi-vehicle convoy acquisition that establishes a genuine cross-trajectory, aerial-to-ground NVS benchmark for autonomous driving.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously benchmarks 8 NVS models across multiple baselines and view disparities, complemented by depth filtering, dynamic masking, and feed-forward pose estimation analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Lucid formulation of the extrapolation bottleneck, well-structured mathematical verification of poses, and thorough empirical reporting.
  • Value: ⭐⭐⭐⭐⭐ Highly impactful benchmark for driving simulation fidelity, neural radiance rendering, and wide-baseline visual localization.