title: >- [Paper Note] When the City Teaches the Car: Label-Free 3D Perception from Infrastructure description: >- [ECCV 2026][autonomous_driving][3D Object Detection] Proposes an infrastructure-taught, label-free 3D perception paradigm where stationary RSUs unsupervisedly learn 3D detectors via temporal persistence and broadcast pseudo-labels to train a standalone ego detector without manual labels or test-time V2X communication. tags: - ECCV 2026 - autonomous_driving - 3D Object Detection - Label-Free Learning - V2X date: 2026-09-19 content_hash: 789d0a660d89e888

When the City Teaches the Car: Label-Free 3D Perception from Infrastructure¶

Conference: ECCV 2026
Paper: ECCV Official Portal
Code: https://jinsuyoo.info/civet
Area: Autonomous Driving
Keywords: 3D Object Detection, Label-Free Learning, Cooperative Driving, Roadside Infrastructure, Pseudo-Label Distillation

TL;DR¶

This paper introduces an infrastructure-taught, label-free 3D perception paradigm where stationary roadside units (RSUs) learn local 3D detectors from unlabeled LiDAR data via background persistence and broadcast predictions to passing ego vehicles as pseudo-labels, training an ego detector that operates completely standalone at test time with 82.3% vehicle AP on the CARLA-based CIVET benchmark.

Background & Motivation¶

Building robust 3D perception for autonomous driving has long relied on extensive sensory data collection from ego vehicles paired with expensive manual 3D bounding-box annotations. However, traffic distributions, road layouts, and geometric characteristics vary substantially across cities and geographic regions. As deployment expands to new cities, scaling this conventional collect-label-retrain pipeline becomes financially and logistically prohibitive, exposing a fundamental mismatch between finite manual annotation capacity and the open-world complexity of driving environments. Meanwhile, existing unsupervised or self-supervised 3D detection methods remain strictly ego-centric, where severe occlusions, continuous camera and LiDAR motion, and limited sensor range in the ego view make mining clean, reliable pseudo-labels exceedingly difficult.

In parallel, modern urban environments are increasingly instrumented with static roadside units (RSUs) deployed at intersections and along arterial roads to monitor traffic and support cooperative intelligent transportation systems (C-ITS). In contrast to mobile vehicles, RSUs observe traffic scenes from fixed, elevated viewpoints with wide fields of view and constant background geometry. This raises an intriguing opportunity: could the city's own sensing infrastructure act as local perception experts to teach passing autonomous vehicles, drastically reducing manual annotation overhead?

Crucially, shifting the burden of manual annotation from vehicles to thousands of roadside sensors would merely displace the problem rather than solve it. The decisive insight of this work is that RSUs possess a unique property that breaks this impasse: stationarity. Because an RSU observes the exact same physical scene continuously over time, its static background remains temporally constant while dynamic objects (vehicles, cyclists, pedestrians) appear and disappear across frames. This temporal persistence provides a natural, human-label-free signal for discovering moving objects. The core idea is to let stationary roadside units learn local 3D detectors from unlabeled streams using temporal persistence and broadcast their predictions as pseudo-labels to passing vehicles, distilling city-wide supervision into a standalone ego detector that operates at test time without infrastructure dependencies or online V2X communication.

Method¶

Overall Architecture¶

The proposed infrastructure-taught label-free framework organizes perception learning into a three-stage closed loop across offline and online steps: Stage 1 enables stationary RSUs to learn location-specialized 3D detectors offline using temporal consistency without human supervision; Stage 2 performs online broadcasting and spatial aggregation, where passing ego vehicles collect predictions from nearby RSUs within communication range and transform them into the ego reference frame via distance-weighted NMS; Stage 3 trains the ego 3D detector offline on the gathered pseudo-labeled dataset, outputting a standalone vehicle perception model that requires zero roadside communication at test time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Unlabeled Roadside LiDAR Streams"] --> B["Stage 1: Unsupervised RSU Training<br/>Persistence Point Score + Tracking Smoothing"]
    B --> C["Location-Specialized RSU 3D Detectors"]
    C --> D["Stage 2: Label Transfer & Aggregation<br/>Rigid Coordinate Projection + Distance-Weighted NMS"]
    E["Ego Vehicle Driving Observations"] --> D
    D --> F["Unified Ego Pseudo-Labeled Dataset"]
    F --> G["Stage 3: Offline Ego Detector Distillation<br/>CenterPoint / PointPillars Backbone Training"]
    G --> H["Standalone Ego Perception Model<br/>Zero Test-Time V2X or RSU Dependencies"]

Key Designs¶

1. Unsupervised RSU Training: Harnessing Scene Stationarity for Label-Free Object Discovery To eliminate human annotation overhead across roadside infrastructure, Stage 1 exploits the absolute stationarity of RSU viewpoints. Static background structures (road surfaces, buildings, static poles) remain invariant over time, whereas dynamic obstacles exhibit transient spatio-temporal signatures. For each RSU equipped with LiDAR, the system computes the Persistence Point (PP) score across temporally separated frame intervals, filtering out stable background points and designating transient, low-persistence points as object proposal seeds. These points are clustered into coarse 3D boxes via DBSCAN. To mitigate point cloud sparsity variations and bounding box jitter over varying distances, a multi-object tracking module links proposals across time, identifies the frame where the target is closest to the RSU in the horizontal plane (maximizing point density and geometric fidelity), and propagates this reliable bounding box geometry across the entire object tracklet. This produces clean local pseudo-labels to train an RSU-specialized detector (achieving 82.8% average AP across roadside units).

2. Spatial Label Transfer and Distance-Weighted NMS: Robust Multi-Teacher Aggregation When the sensor-equipped ego vehicle traverses the city, it establishes wireless communication with RSUs within a coverage radius (set to a maximum range of 160 meters). Each active RSU executes real-time inference on its local stream and broadcasts predicted 3D bounding boxes $Y_{R_i}$ to the vehicle. The ego vehicle transforms these predictions into its current local coordinate system using the known rigid transformation matrix $T_{R_i \to E}$: $$ \tilde{Y}{R_i} = T) $$ In overlapping multi-RSU coverage zones, duplicate detections and distance-dependent beam divergence introduce noise. The system resolves this using distance-weighted Non-Maximum Suppression (NMS). Specifically, each candidate box is assigned a confidence score inversely proportional to the Euclidean distance between the target object and the emitting RSU: $$ w = \frac{1}{| p_{\text{obj}} - p_{R_i} |_2} $$ This prioritizes detections from closer RSUs that have denser point returns and more accurate surface estimates, suppressing redundant candidates using an IoU threshold of 0.1 and discarding boxes with extreme ranges or insufficient LiDAR point counts to construct a high-precision pseudo-labeled set.}(Y_{R_i

3. Offline Ego Detector Distillation: Decoupled Generalization from Local Specialists Individually, RSU detectors overfit to their specific fixed viewpoint and background geometry, failing to generalize across different intersections or to the ego vehicle's lower, forward-facing perspective (cross-view evaluation drops sharply). Stage 3 synthesizes these geographically dispersed local experts into a single unified perception model. The ego vehicle collects multi-scene sensor observations $X_E$ and spatially aligned pseudo-labels $Y_E$ across its driving trajectories into a comprehensive dataset $D_E = \{(X_E, Y_E)\}$. Standard 3D detection architectures (such as CenterPoint and PointPillars) are trained on $D_E$ offline using standard multi-task detection objectives. Crucially, because distillation occurs entirely offline, the resulting vehicle perception model operates completely independently during inference, requiring no real-time communication, external positioning infrastructure, or active RSUs.

Loss & Training¶

The ego 3D detector and RSU detectors are optimized using standard 3D object detection losses, combining Gaussian focal loss for object center heatmap estimation and Smooth-L1 loss for 3D bounding box regression (dimensions, center offsets, yaw angles, and velocity vectors). Unsupervised RSU detectors use an initial learning rate of $5 \times 10^{-4}$ decayed by a factor of 0.1 at epochs 10 and 15, trained for 20 epochs on NVIDIA A100 GPUs. In Stage 2 broadcasting, realistic transmission imperfections are evaluated with a 100 ms fixed communication delay and Gaussian perturbations on spatial translation ($\sigma = 0.2\text{ m}$) and yaw angle ($\sigma = 0.05\text{ rad}$), paired with heuristic box refinement to compensate for alignment noise.

Key Experimental Results¶

Main Results¶

The framework is evaluated on the CIVET multi-agent benchmark constructed on CARLA and V2Xverse, comprising four towns (Town 1, 2, 7, 10) with 12 RSUs per town (48 RSUs total). In the dense urban environment of Town 10, performance is evaluated using Bird's-Eye-View (BEV) Average Precision (AP) at IoU thresholds of 0.5 for cars and 0.3 for pedestrians and cyclists.

Table 1: Systematic evaluation of infrastructure-taught ego training in Town 10 (reproduced from Paper Table 1)

Ego Detector Backbone	Stage 1 Supervision Source	Stage 1 Method	Comm. Noise	Box Refine	Car AP	Pedestrian AP	Cyclist AP	Average AP
PointPillars	12 RSUs (Unsupervised)	PP score	None	No	74.3	77.0	68.0	73.1
PointPillars	12 RSUs (Unsupervised)	PP score + Tracking	None	No	79.3	77.5	79.0	78.6
PointPillars	12 RSUs (Unsupervised)	PP score + Tracking	Yes	No	80.3	49.1	69.6	66.3
PointPillars	12 RSUs (Unsupervised)	PP score + Tracking	Yes	Yes	76.5	52.7	79.8	69.7
PointPillars	12 RSUs (Supervised)	Ground Truth	None	No	92.1	82.2	92.8	89.0
PointPillars	Ego Ground Truth (Upper Bound★)	Fully Supervised Ego	None	No	94.4	88.0	93.7	92.0
CenterPoint	12 RSUs (Unsupervised)	PP score	None	No	78.7	78.3	61.1	72.7
CenterPoint	12 RSUs (Unsupervised)	PP score + Tracking	None	No	82.3	79.3	68.5	76.7
CenterPoint	12 RSUs (Unsupervised)	PP score + Tracking	Yes	No	80.4	52.3	66.4	66.4
CenterPoint	12 RSUs (Unsupervised)	PP score + Tracking	Yes	Yes	77.7	56.0	75.7	69.8
CenterPoint	12 RSUs (Supervised)	Ground Truth	None	No	93.9	84.4	93.4	90.6
CenterPoint	Ego Ground Truth (Upper Bound★)	Fully Supervised Ego	None	No	94.4	91.8	96.6	94.3

Ablation Study¶

Table 2: Geographic scalability across four combined towns with 48 RSUs (reproduced from Paper Table 4)

Ego Detector Backbone	Stage 1 Supervision Source	Comm. Noise	Box Refine	Car AP	Pedestrian AP	Cyclist AP	Overall Avg. AP
PointPillars	48 RSUs (Unsupervised)	None	No	84.1	78.4	80.2	80.9
PointPillars	48 RSUs (Unsupervised)	Yes	No	80.4	56.8	77.8	71.7
PointPillars	48 RSUs (Unsupervised)	Yes	Yes	81.7	63.9	81.7	75.8
PointPillars	4-Town Ego Ground Truth (Upper Bound)	None	No	92.4	89.3	94.6	92.1
CenterPoint	48 RSUs (Unsupervised)	None	No	82.7	81.0	78.3	80.7
CenterPoint	48 RSUs (Unsupervised)	Yes	No	78.2	53.5	72.5	68.1
CenterPoint	48 RSUs (Unsupervised)	Yes	Yes	80.5	66.4	81.5	76.1
CenterPoint	4-Town Ego Ground Truth (Upper Bound)	None	No	91.0	90.6	95.3	92.3

Table 3: Complementarity with ego-centric unsupervised methods using PointRCNN on dynamic objects (reproduced from Paper Table 5)

Supervision Source	Communication Noise	Dynamic Object AP	Analysis & Observations
MODEST (Ego-centric Unsupervised)	–	18.0	Baseline ego-motion persistence clustering
Oyster (Ego-centric Unsupervised)	–	39.8	SOTA ego-centric contrastive tracking baseline
Ours (Infrastructure-Taught)	None	62.0	Substantially outperforms ego baselines (+22.2 AP over Oyster)
Ours (Infrastructure-Taught)	Yes	49.9	Retains superior accuracy (+10.1 AP over Oyster) under noise
Oyster + Ours (Joint Combination)	None	64.9	Fused supervision achieves the highest detection accuracy
Oyster + Ours (Joint Combination)	Yes	59.5	Robustness under communication noise improves markedly (+9.6 AP)

Key Findings¶

Infrastructure supervision closely matches supervised upper bounds: With CenterPoint, label-free RSU supervision achieves 82.3% Car AP and 76.7% Average AP in Town 10, recovering over 81% of the fully supervised ego upper bound (94.3% Avg AP) and closely rivaling supervised RSU supervision (90.6% Avg AP).
Pedestrian detection exhibits heightened sensitivity to communication noise: Under simulated transmission delay and coordinate noise, Car AP drops only slightly from 82.3% to 80.4%, whereas Pedestrian AP plummets from 79.3% to 52.3%. The smaller physical volume and sparse LiDAR returns of pedestrians make them vulnerable to slight spatial misalignments. Applying heuristic box refinement restores Pedestrian AP back to 56.0% (and to 52.7% on PointPillars).
RSU density and intersection layouts yield compounding gains: Scaling RSU availability from 4 to 8 and 12 units produces steady AP gains. Comparing placement geometries under a fixed 4-RSU budget reveals that intersection placement (Car/Ped/Cyc AP of 56.9/37.2/46.3) substantially outperforms linear (36.1/15.3/12.9) and corner (46.7/17.9/12.6) placements due to high traffic diversity and balanced spatial coverage.
Effective zero-label cross-domain adaptation: A model pretrained in rural Town 7 drops to 59.2% Car AP when deployed directly in urban Town 10. Fine-tuning using urban infrastructure-taught pseudo-labels elevates Car AP to 81.9% and Cyclist AP to 76.1% (surpassing the 68.5% of training from scratch), confirming strong transferability without manual labels.

Highlights & Insights¶

Decoupled temporal-spatial V2X paradigm: Traditional collaborative perception couples vehicle safety to low-latency wireless communication and live infrastructure availability at inference time. This paper decouples the interaction by confining V2X exchange strictly to offline training pseudo-label collection, leaving the vehicle completely self-contained and resilient during deployment.
Breaking the circular annotation dilemma: By targeting the stationarity of roadside sensors, the authors eliminate the need to annotate roadside datasets manually. Fixed background persistence allows automated unsupervised object discovery, creating a truly label-free end-to-end data flywheel.
Orthogonal complementarity with ego-centric self-supervision: Vehicle sensors excel in near-range point cloud density, whereas roadside sensors provide elevated, occlusion-free coverage of complex interactions. Combining infrastructure pseudo-labels with ego-centric self-supervision (Oyster) boosts noisy dynamic object detection from 49.9% to 59.5% AP, demonstrating compelling synergy.

Limitations & Future Work¶

Reliance on dynamic motion cues: Object discovery via PP scores depends heavily on motion contrast against static backgrounds. Static foreground obstacles (e.g., parked vehicles, stationary debris) remain undetectable in the initial stage, necessitating integration of semantic or multimodal vision-language priors.
Viewpoint disparity and occlusion shadows: The elevated, steep pitch of RSU LiDAR beams contrasts with the horizontal scanning pattern of vehicle-mounted sensors. In some occluded zones, projected RSU boxes may enclose regions containing zero ego LiDAR returns, requiring ray-casting visibility checks during pseudo-label filtering.
Simulation-to-real transfer challenges: The study is validated on CARLA/V2Xverse. Real-world deployment will encounter physical RSU mounting vibrations, sensor calibration drift over temperature cycles, severe weather degradation, and diverse unstructured traffic dynamics.

vs. Online Collaborative Perception (e.g., V2X-Sim, RCooper, DAIR-V2X): Conventional V2X perception streams high-bandwidth neural features or raw points in real time to perform multi-agent fusion during driving, failing when network connectivity drops. In contrast, this work uses infrastructure solely during training to distill a standalone ego detector that requires zero runtime communication.
vs. Ego-Centric Unsupervised 3D Detection (e.g., MODEST, Oyster, LISO): Ego-centric self-supervised methods struggle to disentangle sensor ego-motion from dynamic object movement, leading to noisy clusters. Leveraging stationary roadside viewpoints provides superior temporal object isolation, achieving an immediate 22.2 AP jump over Oyster on dynamic objects.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates an innovative paradigm that leverages stationary roadside infrastructure as unsupervised teachers to train standalone ego vehicles.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation spanning 4 towns, 48 RSUs, communication noise stress testing, placement ablation, and domain transfer.
Writing Quality: ⭐⭐⭐⭐⭐ Clear structure with intuitive three-stage formulation addressing the core bottleneck of manual 3D annotation costs.
Value: ⭐⭐⭐⭐⭐ Establishes a highly practical, label-free data-engine framework uniting smart city infrastructure with scalable autonomous driving perception.