CoGoal3D: Collaborative 3D Object Detection with 3D-Aware Fusion and Refinement¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Autonomous Driving
Keywords: Collaborative Perception, 3D Object Detection, 3D Spatial Alignment, Point Cloud Reconstruction, Data Augmentation

TL;DR¶

To tackle 3D spatial misalignment caused by disparate heights and attitudes among collaborators in V2X perception, CoGoal3D introduces multiscale 3D-Aware Global Fusion (3D-AGF) and Reconstruction-Guided Local Refinement (RGLR) coupled with Multi-Agent Collaborative Data Augmentation (MCDA), setting new SOTA performance across real-world datasets.

Background & Motivation¶

Environmental perception serves as the cornerstone of autonomous driving. However, single-vehicle perception systems are fundamentally bounded by restricted sensor horizon ranges and visual occlusions, frequently suffering detection failures in complex urban environments like intersections. In response, Vehicle-to-Everything (V2X) and Vehicle-to-Vehicle (V2V) collaborative perception frameworks have emerged as promising solutions by aggregating multi-perspective features across distributed agents. Mainstream collaborative perception methods adopt intermediate feature fusion built on a broadcast communication paradigm: each collaborating agent locally extracts Bird's-Eye-View (BEV) features and broadcasts them alongside its 6-DoF pose to the ego-vehicle, which then projects the received BEV features onto its own coordinate frame via 2D BEV warping before fusion.

Nevertheless, conventional 2D BEV warping implicitly assumes that all collaborative agents observe the environment on an identical horizontal plane. In real-world V2X and V2V scenarios, roadside infrastructure LiDARs are mounted on elevated poles (causing immense vertical height discrepancies), while moving vehicles experience continuous roll and pitch variations due to suspensions, varying loads, and uneven road topography. Neglecting collaborator height and attitude differences during planar 2D BEV warping creates severe spatial misalignment when projected into actual 3D space. While such errors may be somewhat tolerated in coarse 2D BEV detection, they drastically degrade the height and orientation accuracy required for genuine 3D bounding box detection. Furthermore, existing collaborative data augmentation methods (such as DPTP) apply global rotations and scalings in the unified ego-vehicle frame, which regularly shifts collaborators' point clouds outside their valid sensory detection zones, causing significant information loss.

To resolve these critical bottlenecks without sacrificing broadcast communication efficiency, this work focuses on addressing spatial misalignment and collaborative data augmentation concurrently. Core idea: Build a two-stage progressive 3D alignment and refinement pipeline that couples multiscale 3D-aware global fusion (3D-AGF) with reconstruction-guided local refinement (RGLR) under ground-truth optimization, powered by zero-information-loss multi-agent collaborative data augmentation (MCDA).

Method¶

Overall Architecture¶

CoGoal3D adopts a broadcast-based two-stage collaborative perception architecture. The input point clouds from the ego-vehicle and collaborating agents are encoded into BEV features using shared PointPillars backbones. In Stage 1, the ego-vehicle receives the collaborator features and poses, applying the multiscale 3D-Aware Global Fusion (3D-AGF) module to incorporate 3D position encodings and 3D-aware deformable cross-attention, generating candidate 3D proposals via a Region Proposal Network (RPN). In Stage 2, proposal features extracted via BEV RoI pooling enter the Reconstruction-Guided Local Refinement (RGLR) module. Here, a refined 3D detection head operates in parallel with an auxiliary 3D point cloud reconstruction head supervised by ground-truth optimized (GTO) point clouds. Training is bolstered by the Multi-Agent Collaborative Data Augmentation (MCDA) strategy.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Agent Input LiDAR Points & Poses"] --> B["Multi-Agent Collaborative Data Augmentation (MCDA)<br/>Local GT Rotation + Global Scaling & Independent Flipping"]
    B --> C["Shared Point Cloud Backbone (PointPillars)<br/>Extract Multi-Scale BEV Features"]
    C --> D["Multiscale 3D-Aware Global Fusion (3D-AGF)<br/>3D Position Encoding + 3D Deformable Cross-Attention"]
    D --> E["RPN Proposal Generation<br/>Multiscale Fused Features ➜ Initial 3D Proposals"]
    E --> F["Reconstruction-Guided Local Refinement (RGLR)<br/>BEV RoI Pooling ➜ Self-Attention Interaction"]
    F --> G["Parallel Detection & Reconstruction Heads<br/>Refined 3D Bounding Boxes + Auxiliary 3D Point Reconstruction (GTO Supervised)"]

Key Designs¶

1. Multiscale 3D-Aware Global Fusion (3D-AGF): Overcoming Elevation and Attitude Misalignment

Standard broadcast methods align BEV features using rigid 2D planar warping, completely discarding 3D geometric shifts induced by pole-mounted infrastructure heights and vehicular pitch angles. 3D-AGF addresses this spatial misalignment in two consecutive steps. First, it injects explicit 3D position encodings. Given the spatial transformation matrix $\Gamma_{j \to i}$ between collaborator $j$ and ego $i$, the 3D coordinates of collaborator pillar centers at scale $l$, denoted $p_{j,l} = (x_{j,l}, y_{j,l}, z_{j,l})$, are mapped to the ego frame: $$p_{j\to i, l} = \Gamma_{j\to i} p_{j,l}$$ The transformed coordinates are encoded via an MLP and added directly to collaborator BEV feature $F_{j,l}$. Second, 3D-aware deformable cross-attention is deployed. Taking each ego BEV grid feature as query $q_{i,l}$, its 3D pillar center is back-projected onto the collaborator's coordinate space via $\Gamma_{i\to j}$ to obtain a 3D-aware reference point $r_{q,l}$ on the collaborator's plane. Deformable attention then learns data-dependent sampling offsets around $r_{q,l}$, dynamically adapting to non-rigid spatial distortions caused by attitude offsets before aggregating the aligned features into the ego representation.

2. Reconstruction-Guided Local Refinement (RGLR) and Ground Truth Optimization (GTO): Geometric Regularization via Supervised Point Reconstruction

To recover fine-grained 3D geometric boundaries flattened during 2D BEV compression, Stage 2 introduces an auxiliary 3D point cloud reconstruction task. For each 3D proposal $b_m$, BEV RoI pooling extracts candidate feature grids $G \times G$. A relative 3D position encoding is computed for each grid pillar center $p_g$ with respect to proposal center $r_c$ and its 8 bounding box vertices $r_1, \dots, r_8$: $$p_g^{pos} = \text{MLP}([p_g - r_c; p_g - r_1; \dots; p_g - r_8])$$ After feature enhancement via self-attention, the reconstruction head predicts a set of $N_p$ local 3D surface points per pillar grid. To supervise this reconstruction, the Ground Truth Optimization (GTO) module matches collaborator ground-truth boxes to ego boxes using the Hungarian algorithm based on 3D IoU (or track IDs). It computes the fine relative transformation between matched pairs and transforms collaborator points into the ego box frame, eliminating ghosting artifacts caused by sensor calibration tolerances and temporal latency.

3. Multi-Agent Collaborative Data Augmentation (MCDA): Zero-Information-Loss Multi-Agent Perturbation

In single-vehicle perception, global rotations and flips are standard practice. However, applying naive global rotations across all agents in the ego coordinate system (as done in DPTP) often rotates collaborator point clouds outside their physical detection boundaries, causing acute data loss. MCDA restructures the augmentation sequence: 1. Unified Local GT Rotation: Point clouds from all collaborators are transformed into the ego frame, where random rotations ($[-\pi/20, +\pi/20]$) are applied strictly around each ground-truth object's center, diversifying object yaw angles while keeping global positions intact; 2. Unified Global Scaling: A slight uniform scaling ($[0.95, 1.05]$) expands the training diversity; 3. Independent Global Flipping: Collaborator points are reprojected back to their individual sensor coordinate frames and flipped along the $x$-axis alongside their pose vectors. Because sensor detection envelopes are symmetric, this prevents targets from being rotated out of valid bounds.

Loss & Training¶

The entire network is trained end-to-end with a composite multi-task objective: $$L_{\text{total}} = L_{\text{RPN}} + L_{\text{refine}} + L_{\text{rec}}$$ $L_{\text{RPN}}$ combines Focal Loss for classification and Smooth L1 Loss for 3D anchor regression. $L_{\text{refine}}$ combines Binary Cross-Entropy (BCE) loss for proposal confidence and Smooth L1 Loss for bounding box delta refinement. The auxiliary reconstruction objective $L_{\text{rec}}$ computes the Chamfer Distance (CD) between reconstructed point predictions and GTO-optimized ground-truth points. The model is trained on an NVIDIA RTX 3090 GPU using the Adam optimizer for 60 epochs with an initial learning rate of 0.001.

Key Experimental Results¶

Main Results¶

CoGoal3D was evaluated against prior state-of-the-art methods across DAIR-V2X (validation set), V2V4Real (test set), and V2X-Real (test set).

Performance on DAIR-V2X Validation Set (Source: Paper Table 1 excerpt):

Method	Comm. Type	BEV [email protected]	BEV [email protected]	3D [email protected]	3D [email protected]	Inference FPS
No Fusion (Single-Vehicle)	-	65.56	53.89	59.65	29.44	32.4
DiscoNet (NeurIPS 2021)	Broadcast (B)	73.52	58.15	64.01	32.34	29.7
V2X-ViT (ECCV 2022)	Handshake (H)	76.23	58.76	68.68	33.17	16.1
CoAlign (ICRA 2023)	Broadcast (B)	78.14	64.81	68.80	39.69	29.1
DI-V2X (AAAI 2024)	Handshake (H)	79.39	65.39	72.54	39.24	22.3
DSRC (AAAI 2025)	Handshake (H)	74.96	60.23	67.95	36.08	26.1
CoSDH (CVPR 2025)	Broadcast (B)	78.38	64.84	67.95	36.78	6.7
CoGoal3D (Stage 1 Only)	Broadcast (B)	79.49	67.33	73.24	42.66	24.8
CoGoal3D (Full Model)	Broadcast (B)	81.75	72.16	76.59	50.55	16.8

Cross-Dataset Generalization Results (Source: Paper Table 2 excerpt):

Dataset	Metric	DSRC (Prev. SOTA)	CoSDH (Prev. SOTA)	CoGoal3D (Ours)	Absolute Gain
V2V4Real (V2V Benchmark)	BEV [email protected]	53.07	51.25	59.72	+6.65%
	3D [email protected]	21.16	16.67	31.50	+10.34%
V2X-Real (Mixed Multi-Agent)	BEV [email protected]	64.64	70.02	76.98	+6.96%
	3D [email protected]	42.67	25.19	54.64	+11.97%

Ablation Study¶

Systematic ablations were performed on the DAIR-V2X validation set to quantify each architectural module's contribution (Tables 3 & 4):

Core Module Ablation (Source: Paper Table 3):

MCDA	3D-AGF	RCNN	RGLR	3D [email protected]	3D [email protected]	Observations
✕	✕	✕	✕	65.98	33.27	Baseline PointPillars architecture
✓	✕	✕	✕	71.04	40.85	Adding MCDA provides massive +7.58% boost at [email protected]
✓	✓	✕	✕	73.24	42.66	3D-AGF effectively resolves cross-agent 3D misalignment
✓	✓	✓	✕	75.10	48.28	Standard two-stage bounding box refinement
✓	✓	✓	✓	76.59	50.55	Reconstruction-guided head boosts strict 3D [email protected] by +2.27%

Ablation of 3D Position Encoding and Ground Truth Optimization (Source: Paper Table 4):

3D PE	GTO	3D [email protected]	3D [email protected]	Analysis
✓	✓	76.59	50.55	Full pipeline
✕	✓	73.12	46.70	Disabling 3D PE drops 3D [email protected] by 3.85%, highlighting 3D spatial alignment
✓	✕	75.87	49.57	Disabling GTO introduces ghosting noise into reconstruction supervision
✕	✕	73.29	46.99	Neither spatial features nor GT points are corrected

Key Findings¶

3D Spatial Misalignment is the Main Performance Limiter in Collaborative 3D Detection: While prior methods achieve competitive BEV scores, their 3D [email protected] remains below 40% due to planar height/pitch neglect. CoGoal3D's 3D-AGF alone pushes 3D [email protected] to 42.66%, and the full model attains 50.55% (+10.86% over DI-V2X), proving the decisive value of 3D-aware fusion.
Collaborative Augmentation Requires Guarding Against Boundary Truncation: As shown in Table 5, DPTP's global rotations sweep collaborator points out of bounds, causing final 3D [email protected] to drop by 0.5% versus baseline. Conversely, MCDA's local GT rotation strategy reliably increases data diversity without information loss, delivering an 8.67% total gain on 3D [email protected].
Robustness Against Pose Noise and Network Delay: Under varying pose noise ($0\text~~}0.8\text{ m/}^\circ$) and communication latency ($100\text{~~400\text{ ms}$), CoGoal3D's performance degrades much more gracefully than competing methods, supported by deformable sampling offsets and geometric reconstruction regularization.

Highlights & Insights¶

Pairing 3D Position Encodings with Deformable Attention for Collaborative BEV: Rather than forcing rigid 2D planar warping, mapping 3D pillar centers into collaborator space to establish attention reference points elegantly handles physical mounting height differentials and road slope pitch angles.
Ground Truth Optimization (GTO) for Clean Geometric Supervision: Point cloud union across collaborative agents inherently suffers from calibration jitter and temporal asynchrony. GTO's Hungarian-matched local coordinate realignments eliminate ghosting and provide reliable geometric targets for auxiliary reconstruction.
Object-Centric Local Rotation for Multi-Agent Data Augmentation: Resolves the subtle yet pervasive failure mode in multi-agent data pipelines where naive global rotations inadvertently cast distant collaborator points outside sensory coverage bounds.

Limitations & Future Work¶

Reliance on External Pose Priors: While deformable attention compensates for moderate pose jitter, the calculation of initial 3D reference points still requires reasonably accurate GPS/IMU metadata; severe localization failure could degrade projection quality.
Training Overhead of Point Reconstruction: Calculating Chamfer Distance across proposal grids increases GPU memory usage and training time, though the reconstruction head is discarded during testing without hurting real-time inference (16.8 FPS).

vs Broadcast Methods (CoSDH, CoAlign): Broadcast predecessors warp intermediate BEV representations strictly along the 2D plane, ignoring vertical height and vehicle attitude differences. CoGoal3D incorporates 3D spatial geometry directly into the fusion and refinement phases.
vs Handshake Methods (DI-V2X, DSRC): Handshake models require two communication rounds and localized re-projections, which incur high latency and computational overhead. CoGoal3D achieves superior accuracy within an efficient single-round broadcast framework.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Systematically addresses 3D spatial misalignment in V2X detection via 3D-AGF, GTO, and MCDA]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across DAIR-V2X, V2V4Real, and V2X-Real with rigorous ablations and noise stress testing]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, well-structured mathematical pipeline, and crisp visualizations]
Value: ⭐⭐⭐⭐⭐ [Sets a new benchmark for collaborative 3D perception with practical, real-time-capable engineering designs]