RePL: Pseudo-label Refinement for Semi-supervised LiDAR Semantic Segmentation¶
Conference: ECCV 2026
Paper: ECCV Official
Code: Pending
Area: Autonomous Driving
Keywords: Semi-supervised Learning, LiDAR Semantic Segmentation, Pseudo-label Refinement, Masked Reconstruction, Point Cloud Perception
TL;DR¶
Addressing error propagation and confirmation bias from noisy pseudo-labels in outdoor semi-supervised LiDAR semantic segmentation, RePL detects unreliable voxels via teacher-student agreement and actively reconstructs them with a learnable mask token, setting new state-of-the-art results on nuScenes-lidarseg and SemanticKITTI without per-benchmark hyper-parameter tuning.
Background & Motivation¶
Outdoor 3D LiDAR semantic segmentation provides fundamental environmental perception for autonomous vehicles and intelligent mobile robotics by assigning point-wise semantic category labels. Training high-capacity 3D segmentation architectures conventionally relies on massive point cloud datasets annotated with dense manual labels, which incurs prohibitive human labor and collection expenses. Semi-supervised learning (SSL) serves as a vital solution to mitigate this data collection bottleneck by exploiting a small subset of densely annotated point clouds in conjunction with extensive volumes of unlabeled scans.
Mainstream semi-supervised LiDAR segmentation paradigms predominantly employ consistency regularization across geometric perturbations (e.g., LaserMix, AIScene, IT2) or contrastive feature representations (e.g., GPC, Lim3D, DDSemi). However, these self-training routines suffer from a fundamental vulnerability: confirmation bias toward erroneous pseudo-labels. When the network blindly consumes its own inaccurate predictions as supervisory targets on unlabeled data, initial misclassifications become progressively amplified throughout training iterations, ultimately leading to severe performance stagnation or degradation.
Existing strategies for mitigating noisy pseudo-labels largely rely on post-hoc remedies, such as filtering out predictions below rigid confidence cutoffs or applying confidence-weighted loss scaling. These approaches merely adjust sample utilization or discard unreliable voxels after pseudo-labels are generated, failing to improve pseudo-label quality at the source while discarding informative geometric structures in sparse boundary regions. Core idea: shift the semi-supervised noise management paradigm from post-hoc discarding to active source refinement by identifying potentially erroneous voxels through adaptive teacher-student agreement and reconstructing clean pseudo-labels via masked reconstruction with a learnable token.
Method¶
Overall Architecture¶
The RePL framework integrates two synergistic modules: teacher-student segmentation networks and a dedicated pseudo-label refiner network. Unstructured 3D LiDAR point clouds are voxelized into regular grid representations \(X \in \mathbb{R}^{C \times H \times W \times L}\). The teacher network, updated via exponential moving average (EMA) of the student, generates initial predictions for unlabeled scans. The refiner identifies uncertain voxels where teacher and student disagree or fall below scene-adaptive confidence percentiles. These unreliable positions are masked and substituted with a learnable mask token, then concatenated with raw voxel features and reconstructed by the refiner network. Finally, reliable teacher predictions and reconstructed pseudo-labels are assembled into high-quality supervision for student training using symmetric cross-entropy and scene mixing.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Voxelized Point Cloud X"] --> B["Student Prediction P & Teacher Prediction Q"]
B --> C["Confidence Agreement Error Detection & Random Masking<br/>Compare predicted classes & scene-adaptive percentile thresholds"]
C --> D["Masked Reconstruction Refinement with Learnable Token<br/>Replace uncertain voxels with Token T and reconstruct via Refiner"]
D --> E["Assemble Refined Pseudo-Labels<br/>Combine reliable teacher predictions with reconstructed outputs"]
E --> F["Tri-objective Refiner Training under Sparse Supervision<br/>Ground-truth supervision + Negative learning + LaserMix scene mixing"]
E --> G["Robust Semi-supervised Student Optimization<br/>Symmetric cross-entropy SCE + Lovász-Softmax + Scene mixing"]
F -.->|Update parameters| D
G -.->|EMA momentum update| B
Key Designs¶
1. Confidence Agreement Error Detection & Random Masking: Adaptively identifying error candidates while preventing shortcut memorization
Standard thresholding heuristics rely on rigid global cutoffs, which disproportionately penalize distant and sparse point clouds due to physical beam attenuation. RePL resolves this by combining consensus verification with scene-adaptive relative thresholds. For a given voxel \(\omega\), maximum predicted class probabilities are extracted from both student and teacher models. Adaptive confidence thresholds are dynamically set to the \((100 - \kappa)\)-th percentile of confidence scores across the specific scene (with \(\kappa = 40\%\)). A voxel is deemed reliable strictly when the teacher and student predict identical classes and both confidence scores exceed their respective scene-adaptive thresholds; all other voxels form the candidate error mask \(M = 1 - \mathbf{1}_{\text{rel}}\). To prevent the refiner from merely memorizing localized error artifacts, Bernoulli random masking \(R \sim \text{Bernoulli}(\sigma)\) (\(\sigma = 0.15\)) is injected during refiner training to yield the composite mask \(\tilde{M} = M \lor R\), compelling the refiner to learn global 3D geometric and spatial context.
2. Masked Reconstruction Refinement with Learnable Token: Restoring continuous semantics without discarding geometric topology
Discarding uncertain voxels outright interrupts the geometric continuity of thin objects and ground surfaces. Drawing inspiration from masked autoencoders, RePL addresses error correction through continuous latent space reconstruction. Given teacher prediction tensor \(Q \in \mathbb{R}^{K \times H \times W \times L}\), voxels flagged by \(\tilde{M}\) are replaced with a continuous learnable mask token \(T \in \mathbb{R}^K\):
The mask token \(T\) is optimized end-to-end via gradient descent, serving as a flexible placeholder for missing semantics. The refiner network \(g(\cdot)\) takes the channel-wise concatenation of input point features \(X\) and masked prediction \(\bar{Q}\) to output refined probability distributions \(\hat{Q} = g(X, \bar{Q})\). During pseudo-label generation for student supervision, the random mask is disabled, and refiner outputs substitute only the candidate error positions \(M\), retaining teacher predictions on reliable voxels to form the discrete pseudo-label \(\tilde{Y}\).
3. Tri-objective Refiner Training under Sparse Supervision: Ground-truth supervision, negative learning, and mixed-scene exposure
Because ground-truth annotations are available for only a tiny fraction of point clouds, training the refiner solely on labeled prediction errors risks severe overfitting. RePL establishes a balanced tri-objective training scheme. First, on labeled scans, supervised reconstruction loss \(\mathcal{L}_{\text{sup}}^r\) (cross-entropy and Lovász-Softmax) is calculated exclusively over masked voxels. Second, on unlabeled scans lacking ground truth, negative learning is deployed: the teacher's Top-\(k\) (\(k=3\)) classes are retained as plausible candidates, while non-candidate classes \(\mathcal{N}_j(\omega)\) are penalized via an exclusion cross-entropy loss:
Third, labeled and unlabeled scans are sliced and blended along inclination angles via LaserMix to produce augmented hybrid scenes \(X_m\). By requiring the refiner to reconstruct masked labels on the labeled partitions of hybrid scenes (\(\mathcal{L}_{\text{mix}}^r\)), the refiner is exposed to diverse error distributions, densities, and boundary artifacts. The refiner is optimized by the direct sum \(\mathcal{L}_{\text{sup}}^r + \mathcal{L}_{\text{unl}}^r + \mathcal{L}_{\text{mix}}^r\) without hyper-parameter reweighting.
4. Robust Semi-supervised Student Optimization: Preventing noise overconfidence via bidirectional symmetric cross-entropy
To safeguard the student network against residual noise that survives the refiner, RePL adopts Symmetric Cross-Entropy (SCE) paired with Lovász-Softmax on unlabeled data:
The reverse cross-entropy component \(\mathcal{L}_{\text{ce}}(\tilde{Y}_j, P_j)\) imposes an entropy regularization effect that curbs overconfident fitting to erroneous pseudo-labels. In tandem with mixed-scene student training \(\mathcal{L}_{\text{mix}}^s\), gradients between the student network and the pseudo-label refiner are detached via stop-gradient operations, preventing co-adaptation instabilities during simultaneous optimization.
Theoretical Analysis & Improvement Condition¶
The authors establish formal theoretical guarantees justifying why pseudo-label refinement outperforms de novo generation:
- Refinement Task Difficulty (Proposition 1): Let the original segmentation task be \(Z: X \to Y\) and the refinement task be \(Z': (X, T) \to Y\), where \(T\) denotes teacher guidance. Under conditional entropy formulation, \(H(Y \mid X, T) \le H(Y \mid X)\), proving that refining existing predictions exhibits strictly lower task uncertainty than generating predictions from raw point clouds alone.
- Improvement Condition (Proposition 2): Let \(\pi_j\) denote the precision of the error candidate mask (fraction of actual teacher errors within \(E_j\)). Let \(q_j\) and \(r_j\) represent the correction rate and error introduction rate of the refiner, respectively. Refinement strictly improves accuracy on scene \(j\) if and only if:
Empirical validation on SemanticKITTI reveals that under 1% labeled data (\(\pi = 0.917\)), refinement yields net benefits as long as \(r < 11.05 \cdot q\). Under 50% labeled data (\(\pi = 0.983\)), the margin expands to \(r < 57.8 \cdot q\), confirming that RePL operates well inside the beneficial theoretical regime under realistic perception noise.
Key Experimental Results¶
Main Results¶
RePL was evaluated on the nuScenes-lidarseg (16 classes) and SemanticKITTI (19 classes) benchmarks across four labeled data ratios (1%, 10%, 20%, 50%), utilizing Cylinder3D as the shared backbone.
| Dataset | Label Ratio | Supervised-only | Prev. SOTA (Method) | Ours (RePL) | Gain (vs Sup / vs SOTA) |
|---|---|---|---|---|---|
| nuScenes-lidarseg | 1% | 50.9 | 60.0 (FrustrumMix) | 60.0 | +9.1 / =0.0 |
| nuScenes-lidarseg | 10% | 65.9 | 72.1 (IT2) | 74.4 | +8.5 / +2.3 |
| nuScenes-lidarseg | 20% | 66.6 | 73.5 (IT2) | 75.0 | +8.4 / +1.5 |
| nuScenes-lidarseg | 50% | 71.2 | 74.1 (IT2) | 75.8 | +4.6 / +1.7 |
| nuScenes (Avg.) | - | 63.7 | 69.3 (IT2) | 71.3 | +7.6 / +2.0 |
| SemanticKITTI | 1% | 45.4 | 56.2 (LaserMix++) | 54.7 | +9.3 / -1.5 |
| SemanticKITTI | 10% | 56.1 | 63.3 (AIScene) | 62.5 | +6.4 / -0.8 |
| SemanticKITTI | 20% | 57.8 | 63.7 (AIScene) | 63.2 | +5.4 / -0.5 |
| SemanticKITTI | 50% | 58.7 | 64.9 (FrustrumMix) | 65.9 | +7.2 / +1.0 |
| SemanticKITTI (Avg.) | - | 54.5 | 61.5 (AIScene/FrustrumMix) | 61.6 | +7.1 / +0.1 |
Note: Unlike leading competitors (AIScene, FrustrumMix) that perform bespoke per-dataset hyper-parameter tuning, RePL achieves top average accuracy while keeping all hyper-parameters fixed across both benchmarks. On the weakly supervised ScribbleKITTI benchmark, RePL achieves 56.1% average mIoU, outperforming LaserMix++ (55.4%) and IT2 (55.1%).
Ablation Study¶
1. Loss Term Ablations for Refiner and Student Networks (nuScenes-lidarseg 1%)
| Model Component | Loss Configuration | Improvement Index \(\zeta\) | Val mIoU (%) | Note & Contribution Analysis |
|---|---|---|---|---|
| Refiner Ablation | Sup-only Baseline | - | 50.9 | Base segmentation model |
| Refiner Ablation | \(+\mathcal{L}_{\text{sup}}^r\) (Masked reconstruction on labeled) | 0.327 | 57.2 | Reconstructs labeled errors (+6.3) |
| Refiner Ablation | \(+\mathcal{L}_{\text{sup}}^r + \mathcal{L}_{\text{unl}}^r\) (Negative learning added) | 0.353 | 58.7 | Suppresses unlikely classes on unlabeled data (+1.5) |
| Refiner Ablation | Full Refiner (\(\mathcal{L}_{\text{sup}}^r + \mathcal{L}_{\text{unl}}^r + \mathcal{L}_{\text{mix}}^r\)) | 0.430 | 60.0 | LaserMix hybrid training exposes diverse errors (+1.3) |
| Student Ablation | \(\mathcal{L}_{\text{sup}}^s + \mathcal{L}_{\text{unl}}^s\) (Refined pseudo-labels) | - | 58.1 | Direct training with refined pseudo-labels (+7.2) |
| Student Ablation | \(\mathcal{L}_{\text{sup}}^s + \mathcal{L}_{\text{unl}}^s\) (w/o SCE, standard CE) \(+ \mathcal{L}_{\text{mix}}^s\) | - | 58.0 | Standard CE suffers from confirmation bias (-2.0) |
| Student Ablation | Full Student (\(\mathcal{L}_{\text{sup}}^s + \mathcal{L}_{\text{unl}}^s + \mathcal{L}_{\text{mix}}^s\)) | - | 60.0 | SCE and scene mixing ensure robust convergence |
2. Error Candidate Mask Quality and Mechanism Sensitivity
| Setting / Mechanism | Val mIoU (%) | Key Observation & Takeaway |
|---|---|---|
| Baseline (Teacher w/o Refinement) | 57.0 | Direct performance of initial pseudo-labels |
| Random Mask 25% | 57.6 | Naive random masking provides weak regularization (+0.6) |
| Random Mask 50% | 58.2 | Moderate improvement from spatial masking (+1.2) |
| Random Mask 75% | 58.7 | Excessive masking forces blind regeneration (+1.7) |
| Heuristic Consensus Mask (Ours) | 60.0 | Precise localization of erroneous predictions (+3.0) |
| Oracle Error Mask | 67.3 | Ground-truth error localization upper bound |
| Discard Baseline | 53.5 | Dropping detected unreliable voxels degrades performance below self-training (54.3) |
| w/o Random Masking in Refiner Training | 57.7 | Absence of random masking causes overfitting to localized errors (-2.3) |
Key Findings¶
- Reconstruction substantially outperforms discarding: In the discard-only baseline where unreliable voxels are excluded rather than reconstructed, validation performance drops precipitously to 53.5% mIoU—trailing conventional self-training with raw noisy pseudo-labels (54.3%). In contrast, RePL achieves 60.0% mIoU, demonstrating that active reconstruction preserves structural geometric topology that is otherwise lost.
- Pronounced improvements on rare and small classes: Under the 1% label regime on nuScenes-lidarseg, RePL boosts accuracy across 14 out of 16 categories. Tail categories exhibit dramatic gains: traffic cone (+7.86 IoU), pedestrian (+6.47 IoU), manmade structures (+4.41 IoU), and vegetation (+5.35 IoU), elevating overall pseudo-label mIoU from 59.42% to 62.07%.
- Uniform benefits across sensing ranges: When evaluating pseudo-label quality across radial distance bins (0–10m, 10–20m, 20–30m, 30–40m), RePL achieves consistent improvements of +1.95, +1.30, +1.34, and +1.26 IoU, confirming effectiveness across both dense foreground and sparse distant points.
- Modest computational overhead with backbone versatility: On an NVIDIA RTX 6000 Ada GPU, the refiner adds 0.25 seconds of latency and 396 MB of GPU memory relative to the baseline, yielding a +9.1% mIoU net improvement. Furthermore, when instantiated with the Transformer-based SphereFormer backbone, RePL elevates performance from 50.7% to 56.5% mIoU without architecture-specific re-tuning.
Highlights & Insights¶
- From subtraction to addition in pseudo-label management: While conventional SSL pipelines treat noisy pseudo-labels by discarding or downweighting them, RePL treats noisy regions as corrupted visual tokens and applies continuous masked reconstruction, turning filtering subtraction into constructive semantic restoration.
- Rigorously grounded improvement guarantees: The work proves that refinement possesses strictly lower conditional entropy than generation from scratch, defining the exact closed-form boundary where correction benefits outweigh error introduction.
- High parameter robustness across distinct domains: Unlike methods requiring fine-grained per-dataset tuning of loss weights and mixing thresholds, RePL maintains uniform hyper-parameters across both nuScenes-lidarseg and SemanticKITTI while setting top-tier benchmarks.
Limitations & Future Work¶
- Vulnerability to joint overconfidence: The consensus error detector relies on teacher-student divergence; when both models make identical high-confidence misclassifications due to sensor occlusions or unusual geometries, the erroneous voxels evade refinement.
- Localized over-correction on uniform flat surfaces: Fine-grained per-class breakdowns show minor regressions on extensive continuous classes such as drivable surface (-1.47 IoU) and sidewalk (-2.40 IoU) due to sporadic over-correction.
- Future directions: Replacing the heuristic consensus mask with an end-to-end learnable Error Localization Network (ELN) holds strong promise to narrow the gap toward the 67.3% mIoU Oracle upper bound.
Related Work & Insights¶
- vs LaserMix / AIScene / FrustrumMix: Prior methods design sophisticated data mixing techniques (e.g., spatial beams, frustum cones) to enforce perturbation consistency, but treat raw model predictions as uncorrected ground truth. RePL incorporates LaserMix while introducing an active refiner network to clean the pseudo-supervisory signal itself.
- vs Filtering Baselines (FixMatch / GPC / Lim3D): Hard filtering discards low-confidence points and disrupts continuous 3D geometric surfaces; RePL demonstrates that token-based masked reconstruction yields a +6.5% mIoU advantage over discarding.
- vs ST++ / ELN in 2D Vision: While 2D error localization has focused on image patch selection or iterative re-training, RePL pioneers tokenized masked pseudo-label reconstruction within sparse, unstructured 3D LiDAR voxel representations.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Introduces masked reconstruction with learnable tokens for pseudo-label correction in semi-supervised 3D LiDAR segmentation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across two primary outdoor benchmarks and weak scribble annotations, complete with theoretical proofs, ablations, range analysis, and alternative backbones.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulations, clear pedagogical progression, and well-structured empirical analysis.
- Value: ⭐⭐⭐⭐⭐ Delivers an actionable, plug-and-play paradigm for deploying highly accurate 3D scene perception under limited annotation budgets in autonomous driving.