CSOT: Cross-Scan Object Transfer for Semi-Supervised LiDAR Object Detection

Conference: ECCV 2024
Code: https://github.com/JinglinZhan/CSOT
Area: Autonomous Driving / 3D Object Detection
Keywords: Semi-Supervised Learning, LiDAR Object Detection, Cross-Scan Object Transfer, Point Cloud Data Augmentation, Partial Annotation

TL;DR

The CSOT (Cross-Scan Object Transfer) paradigm is proposed, which predicts semantically consistent object placement locations and compatibility scores using a Transformer network. This achieves the first successful object copy-paste augmentation in semi-supervised LiDAR object detection. Combined with a spatial-aware classification loss, it matches the performance of the fully supervised baseline using only 1% of the annotated data.

Background & Motivation

Background: 3D LiDAR object detection is a core task in autonomous driving perception, but large-scale 3D bounding box annotation is extremely costly (annotating one frame of a point cloud takes more than 10 times longer than a 2D image). Semi-supervised object detection (SSOD) alleviates the annotation bottleneck by leveraging large amounts of unlabeled data, representing a current research hotspot.

Limitations of Prior Work: Mainstream pseudo-labeling methods in LiDAR SSOD face severe challenges: (1) pseudo-labels generated by the teacher model on unlabeled data are highly noisy, requiring meticulous hyperparameter tuning (e.g., confidence thresholds); (2) the sparsity and occlusion of 3D point clouds make it harder to guarantee pseudo-label quality; (3) under extremely low annotation rates (e.g., 1-5%), the teacher model itself lacks sufficient performance, making pseudo-labels almost unusable. Another strategy is data augmentation, but mature 2D copy-paste augmentation is difficult to apply directly in LiDAR—simply copying object point clouds from one scene to another causes physical inconsistencies (e.g., objects floating in the air, embedded in walls, or appearing in unrealistic locations).

Key Challenge: Object placement in LiDAR scenes is subject to strict physical constraints (must be on the ground, cannot overlap with other objects, and must conform to traffic scene logic). Simple random copy-paste cannot satisfy these constraints, while pseudo-labeling methods are unreliable under low annotation rates.

Goal: (1) Design an intelligent object placement mechanism in LiDAR scenes to make copy-paste augmentation feasible in point clouds; (2) handle the false-negative problem (unlabeled objects treated as background during training) in partially annotated scenes; (3) construct a LiDAR SSOD framework that does not rely on pseudo-labels.

Key Insight: The authors propose to reformulate the copy-paste problem as an "object placement prediction" problem, training a dedicated Transformer network to predict suitable locations for placing specific categories of objects in unlabeled scenes, alongside predicting placement compatibility scores. This delegative approach solves physical inconsistency via a learnable network.

Core Idea: Use a Transformer network to predict reasonable placement locations for objects in unlabeled LiDAR scenes, making copy-paste augmentation feasible for the first time in LiDAR SSOD, and handle false negatives with a spatial-aware loss.

Method

Overall Architecture

The CSOT framework consists of three core components: (1) Object Placement Network (OPN)—a Transformer network that takes unlabeled LiDAR scene point clouds as input and outputs the compatibility score of object placement for each position in the scene; (2) Cross-Scan Object Transfer—which extracts object point clouds from labeled data and places them into the optimal positions of unlabeled scenes according to OPN predictions, generating partially labeled training scenes; (3) Spatial-Aware Classification Loss—a specially designed loss function to handle the presence of unlabeled ground-truth objects (false negatives) in partially labeled scenes. The entire workflow does not rely on pseudo-labels, but rather maximizes the utilization of unlabeled data through augmented data.

Key Designs

1. Object Placement Network (OPN):

   • Function: Predicts reasonable placement positions and compatibility scores for objects in unlabeled LiDAR scenes
   • Mechanism: The OPN receives a frame of unlabeled LiDAR point cloud and first extracts BEV (bird's-eye view) feature maps via a 3D backbone network (such as VoxelNet or PointPillars). A Transformer encoder-decoder architecture is then used to process the BEV features. The encoder captures global scene context (road layout, existing object distribution, drivable areas, etc.), and the decoder uses a set of learnable queries to predict candidate placement positions in the scene. Each query outputs a position coordinate \((x, y, z, \theta)\) and a compatibility score \(s \in [0, 1]\), indicating the appropriateness of placing an object at that position. During training, the ground-truth locations of objects in labeled scenes are used as supervision signals to let OPN learn "what kinds of locations are suitable for what categories of objects"
   • Design Motivation: Traditional copy-paste random placement leads to physical inconsistencies. OPN ensures that placement positions comply with physical constraints and traffic scene logic by learning scene semantics

2. Cross-Scan Object Transfer Strategy:

   • Function: Organically transfers object point clouds from labeled data into unlabeled scenes
   • Mechanism: An object point cloud bank (Object Bank) is maintained from labeled data, categorized by class. For each unlabeled scene, OPN predicts \(K\) candidate positions and their compatibility scores, selecting the top-\(N\) positions ranked by score. Corresponding classes of object point clouds are then randomly sampled from the Object Bank and placed into the selected positions after coordinate transformation (translation, rotation, ground height alignment). Occlusion relationships of objects are also handled during placement—deciding the occlusion order based on distance and appropriately removing points in occluded regions. The final generated scene contains the original unlabeled point cloud plus the transferred labeled objects, forming a "partially annotated" training sample
   • Design Motivation: This strategy transfers label information from labeled scans to unlabeled scans, effectively "borrowing" existing labels to expand training data and avoiding the noise issues associated with pseudo-labels

3. Spatial-Aware Classification Loss:

   • Function: Addresses the false-negative problem in partially annotated scenes
   • Mechanism: In partially annotated scenes generated by CSOT, unlabeled ground-truth objects may exist in the original scene alongside the transferred labeled objects. Following standard training pipelines, these unlabeled objects would be treated as background (negative samples) during training, leading to severe false-negative misguidance. The Spatial-Aware Classification Loss addresses this by spatially distinguishing between "confirmed background" and "potential object" regions: for open areas far from labeled objects, the classification loss is computed normally; for areas where the detector predicts high confidence but lack annotations, the negative sample loss weights are reduced or ignored. Specifically, a confidence heatmap is defined on the BEV space, and negative sample losses in high-confidence regions are decayed
   • Design Motivation: Partial annotation is an inherent issue of the CSOT paradigm, necessitating a dedicated loss design to prevent false negatives from misleading the training

Loss & Training

The overall training is divided into two stages: (1) OPN pre-training—using all labeled data to train the OPN to learn object placement capabilities; (2) detector training—using the CSOT-augmented data to train the 3D detector, where the loss function includes standard 3D detection losses (classification + regression + direction) plus the Spatial-Aware Classification Loss. OPN training uses Hungarian matching to pair predicted positions with ground-truth object positions, with losses including position regression loss and compatibility classification loss.

Key Experimental Results

Main Results

Dataset	Annotation Rate	Metric	Ours (CSOT)	Prev. SOTA	Gain
Waymo	1%	mAPH L2	Close to fully supervised	3DIoUMatch, etc.	Significantly outperforms pseudo-labeling methods
Waymo	5%	mAPH L2	Exceeds fully supervised 1%	Pseudo-labeling SSOD	State-of-the-art semi-supervised results
Waymo	20%	mAPH L2	Close to 100% annotations	Various label-efficient methods	Most efficient annotation utility
nuScenes	5%	NDS	99% of fully supervised	Various semi-supervised methods	Nearly perfect reproduction of full supervision
nuScenes	10%	NDS	Exceeds fully supervised	Various semi-supervised methods	Advantage of augmented data diversity

Most notable result: Using only 1% of the annotated data on the Waymo dataset, the performance of CSOT's semi-supervised detector matches the fully supervised baseline. This was previously inconceivable for LiDAR SSOD methods.

Ablation Study

Configuration	Key Metric	Description
Random placement vs. OPN placement	mAPH differs by ~5%	Intelligent placement is far superior to random placement
W/o Spatial-Aware Loss	Performance drops by ~3%	False negative issue severely degrades training
Different number of transferred objects N	N=15-20 is optimal	Excessive objects cause unrealistic crowding
OPN prediction accuracy	>85% IoU matching rate	OPN accurately predicts reasonable locations
CenterPoint vs. PointPillars	Improvement in both	CSOT framework is backbone-agnostic

Key Findings

• CSOT's advantage is most pronounced under extremely low annotation rates, outperforming concurrent pseudo-labeling methods by over 10 percentage points at 1% annotations.
• The placement patterns learned by OPN possess logical semantics—vehicles are primarily placed on roads and pedestrians on sidewalks, aligning with traffic scene common sense.
• The Spatial-Aware Loss becomes more critical as the annotation rate decreases because the proportion of false-negative objects in unlabeled scenes is higher.
• CSOT is complementary to pseudo-labeling methods—the two can be integrated to further improve performance.
• Under the nuScenes 5% annotation setting, CenterPoint + CSOT achieves 99% of the fully supervised CenterPoint's NDS score.

Highlights & Insights

1. Paradigm Innovation: Replaces pseudo-labeling with data augmentation, avoiding the root issue of poor pseudo-label quality, presenting a clean and effective idea.
2. Ingenious OPN Design: Reformulates the satisfaction of physical constraints into a learnable prediction problem, which is more flexible and accurate than hand-crafted rules.
3. Breakthrough under Extremely Low Annotation Rates: Matching fully supervised performance with only 1% annotations is a highly impressive result, which is of great significance for autonomous driving scenarios where annotation costs are extremely high.
4. Framework Generality: Does not bind to a specific detector and can cooperate with various detectors such as CenterPoint and PointPillars.
5. Introduction of Spatial-Aware Loss: Provides an elegant solution to the false-negative training issue in partially annotated scenes.

Limitations & Future Work

1. OPN requires an additional pre-training step, increasing the complexity of the training pipeline.
2. Occlusion handling after object transfer is relatively simple (distance-based point removal); physical realism can be further enhanced.
3. Currently, object transfer is only considered for single-frame point clouds, lacking the use of temporal information for more coherent scene construction.
4. The diversity of objects in the Object Bank is limited by the amount and category distribution of labeled data.
5. Comparisons with recent weakly supervised and active learning methods are missing.
6. The LiDAR intensity of transferred objects may mismatch that of the target scene.

Related Work & Insights

• 2D Copy-Paste Augmentation: Simple Copy-Paste (CVPR 2021) has achieved great success in instance segmentation, but direct transfer to 3D faces additional physical constraints.
• LiDAR SSOD: Methods like 3DIoUMatch and Pseudo-Label are currently mainstream; CSOT provides a new path independent of pseudo-labels.
• Point Cloud Data Augmentation: Methods like GT-Aug also utilize object replication but are limited to fully supervised settings with random placement. CSOT achieves intelligent placement through OPN.
• Scene Generation: Relates to point cloud simulation methods such as LiDARsim and UniSim, but CSOT is more lightweight and directly usable for training.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First to implement intelligent copy-paste in LiDAR SSOD, presenting paradigm innovation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Very comprehensive across Waymo + nuScenes, with multiple annotation rates and detectors.
• Writing Quality: ⭐⭐⭐⭐ Clear problem definition and well-structured method descriptions.
• Value: ⭐⭐⭐⭐⭐ Breakthrough results under extremely low annotation rates, of great significance for cost reduction in autonomous driving annotations.

Related Papers

• [ECCV 2024] ItTakesTwo: Leveraging Peer Representations for Semi-supervised LiDAR Semantic Segmentation
• [ECCV 2024] Weakly Supervised 3D Object Detection via Multi-Level Visual Guidance
• [ECCV 2024] Equivariant Spatio-Temporal Self-Supervision for LiDAR Object Detection
• [ECCV 2024] Detecting As Labeling: Rethinking LiDAR-camera Fusion in 3D Object Detection
• [ECCV 2024] OPEN: Object-wise Position Embedding for Multi-view 3D Object Detection