VIPS: Vehicle-Infrastructure Cooperative Planning Benchmark via Pseudo-Simulation

Conference: ECCV 2026
Paper: ECCV Official
Code: https://vips2026.github.io
Area: Autonomous Driving
Keywords: Cooperative Driving, End-to-End Autonomous Driving, Pseudo-Simulation, Sparse Representation, Motion Planning

TL;DR

Introduces VIPS, the first pseudo-simulation benchmark for vehicle-infrastructure cooperative planning evaluating robustness and error recovery via synchronized real and perturbed novel-view observations, paired with CoS-V2X, an efficient sparse planning framework.

Background & Motivation

End-to-end autonomous driving maps raw perceptual observations directly into driving trajectories, showing remarkable promise in complex urban traffic environments. However, single-agent systems suffer from bounded sensor range and severe occlusions, frequently leaving blind spots at dense intersections and unprotected turns. Vehicle-to-Infrastructure (V2I) communication addresses this fundamental limitation by leveraging elevated, roadside sensors to provide unobstructed global viewpoints. Despite this promise, evaluating V2I cooperative planning policies faces an inherent dilemma: open-loop evaluation on pre-recorded logs cannot assess compounding errors and recovery from off-policy deviations, while interactive closed-loop simulation suffers from substantial domain discrepancies and exorbitant scaling costs.

Recent pseudo-simulation paradigms in single-agent driving circumvent this trade-off by perturbing trajectory endpoints on real logs to synthesize plausible future states, enabling closed-loop-proxy evaluations without simulator overhead. Yet, expanding pseudo-simulation to cooperative V2I systems presents critical challenges: cooperative decision-making hinges on multi-agent information sharing across heterogeneous viewpoints, demanding strict spatial and temporal consistency when rendering the perturbed ego vehicle across both vehicle and infrastructure cameras simultaneously.

To overcome these barriers, this work investigates how to construct multi-view consistent perturbations directly on real-world cooperative driving data and structure a rigorous two-stage evaluation protocol. The core idea is to establish VIPS, a vehicle-infrastructure cooperative pseudo-simulation benchmark that pairs 3D Gaussian Splatting and instance-level inpainting for multi-agent novel-view synthesis, evaluated via an Extended Predictive Driver Model Score across nominal and perturbed stages, complemented by CoS-V2X, an efficient sparse cooperative planning framework.

Method

Overall Architecture

The VIPS evaluation pipeline operates across two coupled evaluation stages: Stage 1 evaluates nominal driving policies on synchronized real-world vehicle and infrastructure sensory feeds, rolling out trajectories via kinematic vehicle modeling; Stage 2 samples candidate perturbed states around expert trajectory endpoints, synthesizes geometrically consistent paired vehicle (via 3DGS and diffusion enhancement) and infrastructure views (via instance-guided patch retrieval and inpainting), and aggregates multi-sample scores with a distance-based Gaussian kernel. In tandem, the proposed CoS-V2X model achieves efficient cooperative planning through symmetric sparse anchor interaction, exchanging only Top-\(K\) high-confidence instances to drive joint prediction and planning without heavy BEV communication overhead.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Synchronized Real Vehicle & Infrastructure Inputs"] --> B["Stage 1: Nominal Evaluation<br/>Kinematic rollout on real observations"]
    B --> C["Stage 2: Perturbed State Sampling<br/>Hermite spline trajectory & candidate poses"]
    C --> D["Multi-View Viewpoint Synthesis<br/>Vehicle 3DGS + Infrastructure patch-and-fill"]
    D --> E["Unified Benchmark Scoring<br/>EPDMS penalty product & Gaussian aggregation"]
    subgraph CoS_V2X["CoS-V2X Sparse Cooperative Planning Framework"]
        direction TB
        F["Sparse Anchors & Temporal Instance Bank"] --> G["Top-K Bidirectional Cross-Attention Fusion"]
        G --> H["End-to-End Motion Prediction & Parallel Planning"]
    end
    D -.-> CoS_V2X
    CoS_V2X -.-> E

Key Designs

1. Two-Stage V2I Pseudo-Simulation Protocol: Balancing Nominal Quality and Off-Policy Recovery

Standard open-loop metrics cannot measure whether an autonomous vehicle can recover once it drifts away from the nominal path. VIPS introduces a two-stage evaluation protocol: Stage 1 evaluates unperturbed driving behavior on real synchronized sensor feeds, yielding a nominal score \(s_1\) over a 5-second horizon. Stage 2 evaluates recovery capability by sampling perturbations centered at the expert demonstration endpoint, applying lateral offsets (\(\pm 2.0\,\text{m}\) at \(1.0\,\text{m}\) intervals) and longitudinal offsets (up to 7 positions at \(5.0\,\text{m}\) intervals within physically reachable acceleration bounds). For each sampled state, past motion history is constructed using a Hermite spline, and physically invalid states are eliminated through rejection sampling, resulting in 7,168 evaluation samples. The Stage 2 scores are combined using a distance-weighted Gaussian kernel: $\(s_2 = \sum_{i=1}^N \hat{w}_i s_i, \quad \hat{w}_i = \frac{w_i}{\sum_{j=1}^N w_j}, \quad w_i = \exp\left(-\frac{\|x_i - \hat{x}\|^2}{2\sigma^2}\right)\)$ where \(x_i\) represents the starting pose of the \(i\)-th perturbed scenario and \(\hat{x}\) is the predicted endpoint from Stage 1. The unified score is computed as the product \(s_{\text{final}} = s_1 s_2\), penalizing planners that perform well only under unperturbed conditions.

2. Consistent Dual-Perspective Novel-View Synthesis: 3DGS Diffusion Refinement and Infrastructure Patch Inpainting

Cooperative pseudo-simulation requires that relocating the ego vehicle in virtual space yields photorealistic, geometrically aligned views across both onboard and roadside cameras. For onboard vehicle cameras experiencing rapid viewpoint motion, VIPS utilizes 3D Gaussian Splatting (3DGS) tailored for dynamic urban environments, further refined by Diffix3D+ to eliminate neural rendering artifacts and blur. For roadside cameras mounted at fixed, elevated positions where the vehicle occupies a relatively small region, applying 3DGS globally induces severe blur. Instead, VIPS introduces an instance-level patch-and-fill pipeline: it extracts the vehicle patch using SAM3 point-prompt guidance, applies a 3D pose transformation to render the vehicle at the new coordinates, and inpaints the vacated region using background patches retrieved from unoccupied frames of the identical camera. This asymmetric synthesis preserves high structural fidelity across both systems.

3. Unified Multi-Metric Driving Evaluation via EPDMS: Multiplicative Safety and Weighted Quality

To quantify driving competence across diverse traffic rules, VIPS employs the Extended Predictive Driver Model Score (EPDMS). Driving objectives are partitioned into safety-critical violations \(\mathcal{M}_{\text{pen}}\) (No at-fault Collision NC, Drivable Area Compliance DAC, Driving Direction Compliance DDC) and continuous driving quality criteria \(\mathcal{M}_{\text{avg}}\) (Ego Progress EP, Time-to-Collision TTC, Lane Keeping LK, History Comfort HC). The score combines them via multiplicative penalties and weighted averages: $\(\text{EPDMS} = \left(\prod_{n \in \mathcal{M}_{\text{pen}}} \text{score}_n\right) \cdot \left(\frac{\sum_{m \in \mathcal{M}_{\text{avg}}} w_m \cdot \text{score}_m}{\sum_{m \in \mathcal{M}_{\text{avg}}} w_m}\right)\)$ Any critical failure (such as an avoidable collision) sets the corresponding penalty to zero, immediately reducing the aggregate EPDMS to zero and preventing models from trading passenger safety for forward progress.

4. CoS-V2X Sparse Representation Framework: Bandwidth-Efficient Cross-Agent Interaction

Existing cooperative planners rely on dense BEV feature map transmission, incurring prohibitive communication latency and memory footprints. CoS-V2X introduces a sparse anchor-based architecture sharing \(N\) learnable instance queries initialized identically across both agents. The infrastructure perception branch extracts 3D bounding boxes and classification logits, transmitting only the Top-\(K\) high-confidence instance representations \(\mathcal{K}\) to the ego vehicle. The vehicle fuses the received infrastructure hypotheses with its local anchors using bidirectional cross-attention and confidence-based weighting: $\(w_i^s = \frac{z_i^s}{z_i^{\text{veh}} + z_i^{\text{infra}} + \epsilon}, \quad \mathbf{B}_i^{\text{fuse}} = w_i^{\text{veh}}\mathbf{B}_i^{\text{veh}} + w_i^{\text{infra}}\mathbf{B}_i^{\text{infra}}\)$ The fused sparse instance tokens directly feed into the downstream motion prediction and trajectory planning heads without rasterizing back to dense BEV representations, cutting transmission overhead down to \(2.5 \times 10^6\,\text{BPS}\).

Loss & Training

CoS-V2X employs a decoupled multi-task training scheme. The perception modules are trained using Focal Loss for 3D classification, Smooth L1 Loss for 3D bounding box regression, and segmentation losses for vectorized lane mapping. Downstream trajectory prediction and parallel planning heads optimize trajectory classification and imitation regression losses following SparseDrive. The complete architecture is trained end-to-end on 4 NVIDIA A100 GPUs.

Key Experimental Results

Main Results

Evaluation of leading end-to-end driving models on VIPS over a 5-second planning horizon. Results summarize Stage 1 (S1), Stage 2 (S2), and the integrated score (\(S1+S2\)).

Method	V2I Coop.	S1 EPDMS ↑	S2 EPDMS ↑	Integrated EPDMS (S1+S2) ↑	Key Metric Highlights (S2)
Constant Velocity	✗	62.39	68.62	5.88	LK drops to 20.13
AD-MLP [154]	✗	52.73	41.90	32.31	S2 DAC at 63.86
UniAD [36]	✗	75.26	69.11	37.28	S2 LK drops to 25.61
SparseDrive [108]	✗	74.47	55.74	43.26	S2 DDC falls to 68.87
HiP-AD [111]	✗	75.54	68.76	42.03	S2 NC 94.85, TTC 94.33
MomAD [104]	✗	69.95	67.48	45.38	S2 LK 30.68
Uni-V2X [146]	✓	75.70	72.35	43.79	S2 NC 99.10, TTC 97.22
CoS-V2X (Ours)	✓	78.69	73.21	50.88	S2 NC 99.96, DAC 99.98, TTC 99.95

Note: Data from Table 2 of the original paper. Human demonstration provides an S1 reference score of 91.55. CoS-V2X outperforms the prior best cooperative planner Uni-V2X by +7.09 in integrated EPDMS.

Ablation Study

Ablations on cooperative perception gain, systems efficiency, and synthetic novel-view image quality (corresponding to Table 3, Table 4, and Table 6 in the paper).

Evaluation Aspect / Config	Metric	Value	Note
Cooperative Perception (Table 3)	3DOD NDS / mAP ↑ Online Mapping mAP ↑	w/o V2I: 17.78 / 14.63 w/o V2I: 30.46	w/ V2I: 31.30 / 28.13 (+13.50 mAP) w/ V2I: 34.24 (+3.78 mAP)
System Efficiency (Table 4)	Peak Memory (Train / Test) ↓ Inference Speed FPS ↑ Transmission Cost (BPS) ↓	Uni-V2X: 29.38 / 6.95 GB Uni-V2X: 0.2 FPS Uni-V2X: \(3.5 \times 10^6\) BPS	CoS-V2X: 8.86 / 4.81 GB (-69.8% train mem) CoS-V2X: 6.4 FPS (32× speedup) CoS-V2X: \(2.5 \times 10^6\) BPS (-28.6% bandwidth)
Synthetic View Quality (Table 6)	Vehicle View LPIPS ↓ Infrastructure View LPIPS ↓	Standard 3DGS: 0.414 Standard 3DGS: 0.324	Ours (3DGS + Diffix3D+): 0.371 Ours (SAM3 Patch-and-Fill): 0.076

Key Findings

• Perturbation-Induced Performance Cliff: Single-vehicle planners experience severe performance degradation under Stage 2 perturbations (e.g., SparseDrive drops from 74.47 to 55.74), whereas CoS-V2X retains an S2 score of 73.21 with near-perfect safety metrics (NC 99.96, DAC 99.98, TTC 99.95), proving the necessity of infrastructure context for robust recovery.
• Strong Alignment with Expert Human Preferences: In a blind study with 7 expert drivers having over 10 years of driving experience (Table 5), EPDMS achieved a Kendall \(\tau\)-b rank correlation of 0.84 and a pairwise accuracy of 0.94, outperforming the constant velocity baseline (0.23 and 0.62).
• Negligible Synthetic Distribution Gap: Evaluating real-data-trained CoS-V2X on S1 synthetic views reconstructed at original poses (Table 7) revealed minimal drop in 3D detection mAP (28.13 to 26.70) and planning EPDMS (78.69 to 77.60), confirming high photorealism and distribution consistency.

Highlights & Insights

• Scalable Benchmark without Sim-to-Real Gap: VIPS extends pseudo-simulation to cooperative multi-agent autonomous driving for the first time, delivering closed-loop-proxy fidelity on real datasets without relying on game engines.
• Asymmetric Rendering for Multi-Agent Perspectives: Instead of applying heavy 3DGS naively across all views, combining 3DGS on vehicle cameras with SAM3 instance patch relocation on stationary infrastructure cameras yields superior rendering quality (LPIPS 0.076).
• High Efficiency via Sparse Cooperative Interaction: CoS-V2X demonstrates that exchanging only Top-\(K\) sparse instance representations drastically cuts bandwidth and boosts FPS by 32× while outperforming dense BEV fusion models.

Limitations & Future Work

• Non-Reactive Background Agents: Surrounding vehicles follow pre-recorded trajectories in the kinematic rollout, leaving multi-agent feedback responses to ego-vehicle evasive maneuvers unmodeled.
• Reliance on Curated HD Vector Maps: Evaluation requires dense vector maps created by accumulating point clouds on V2X-Real, limiting immediate zero-shot evaluation on unmapped open-road sequences.
• Simplified Network Channel Simulation: While bandwidth is quantified, real-world communication packet drops, latency jitter, and severe asynchronous clock offsets are not modeled dynamically.

Related Work & Insights

• vs NAVSIM v2 [5]: NAVSIM initiated data-driven pseudo-simulation for single vehicles. VIPS pioneers the extension to multi-agent V2I settings, resolving multi-view geometric alignment and roadside rendering challenges.
• vs Uni-V2X [146]: Uni-V2X performs heavy dense BEV feature map aggregation with low throughput (0.2 FPS) and high memory demand. CoS-V2X relies on sparse instance tokens, delivering 6.4 FPS, cutting memory by ~70%, and achieving superior planning scores.
• vs DAIR-V2X / V2X-Seq [143, 145]: Prior V2X benchmarks predominantly focus on perception accuracy and open-loop waypoint errors. VIPS establishes a behavior-centric benchmark spanning safety compliance, rule adherence, and passenger comfort.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First closed-loop-proxy pseudo-simulation benchmark for V2I planning, complemented by asymmetric viewpoint synthesis and sparse cooperative modeling.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive experiments across nominal and perturbed planning stages, perception gains, efficiency profiling, and expert human studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with crisp technical details and rigorous evaluation metrics.
• Value: ⭐⭐⭐⭐⭐ Sets an essential benchmark for evaluating V2I end-to-end planning robustness in real-world scenarios.

Related Papers

• [ECCV 2026] CooperScene: Multi-Modal Cooperative Autonomy Benchmark with C-V2X Communication Characterization
• [NeurIPS 2025] UrbanIng-V2X: A Large-Scale Multi-Vehicle Multi-Infrastructure Dataset Across Multiple Intersections for Cooperative Perception
• [ECCV 2026] RePL: Pseudo-label Refinement for Semi-supervised LiDAR Semantic Segmentation
• [ECCV 2026] DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation
• [ECCV 2026] LiDAR-EVS: Enhance Extrapolated View Synthesis for 3D Gaussian Splatting with Pseudo-LiDAR Supervision