BFANet: Revisiting 3D Semantic Segmentation with Boundary Feature Analysis

Conference: CVPR 2025
arXiv: 2503.12539
Code: https://github.com/weiguangzhao/BFANet
Area: 3D Vision / 3D Semantic Segmentation
Keywords: 3D Semantic Segmentation, Boundary Feature Analysis, Octree, Attention Mechanism, Segmentation Error Classification

TL;DR

Revisiting 3D semantic segmentation from the perspective of error analysis, this study classifies segmentation errors into four categories (region classification, displacement, merge, and false response) and designs corresponding evaluation metrics. It proposes BFANet, which enhances boundary awareness through a boundary-semantic decoupling module and real-time boundary pseudo-label computation, achieving 36.0 mIoU on the ScanNet200 test set (the highest score without utilizing auxiliary data during training).

Background & Motivation

3D semantic segmentation is a fundamental task in 3D scene understanding. Current SOTA methods (such as OctFormer and PTv3) mainly focus on improving overall metrics like mIoU, while ignoring the fine-grained quality analysis of the segmentation. Specifically, they treat all points "equally," which leads to poor performance on challenging regions such as boundaries. This deficiency prevents researchers from deeply understanding where and how models fail. Inspired by frequency analysis in 2D semantic segmentation, this paper is the first to systematically categorize 3D semantic segmentation errors into four types: region classification errors (entire region is misclassified), displacement errors (boundary erosion/dilation), merge errors (other objects are incorrectly merged), and false responses (incorrect regions appearing within semantically connected areas). The analysis reveals that the first three types of errors are closely related to the loss of boundary features. The core idea is to enhance semantic features by explicitly decoupling and utilizing boundary features.

Method

Based on the OctFormer backbone, BFANet incorporates boundary-semantic decoupling and fusion modules, and implements an efficient real-time boundary pseudo-label computation method.

Overall Architecture

The input point cloud is utilized to construct an octree structure, and OctFormer extracts multi-layer features \(f_o\) (leveraging cross-layer interactions of the last four feature layers). Then, the boundary-semantic module decouples \(f_o\) into semantic features and boundary features, and enhances the semantic features by fusing the query sequences of both through an attention mechanism. Finally, the semantic branch and boundary branch output their respective predictions. During training, boundary supervision is provided through boundary pseudo-labels computed online using CUDA parallel programming.

Key Designs

1. Boundary-Semantic Block:

   • Function: Decouples multi-layer features into semantic and boundary features and enhances them through fusion.
   • Mechanism: Employs two independent MLP branches (\(\mathrm{Mb_1}\) and \(\mathrm{Ms_1}\)) to constrain features for obtaining boundary/semantic discriminative power respectively. Then, each branch generates Query/Key/Value triplets through MLPs. The key innovation lies in the fusion mechanism—the boundary Query \(Q_b\) and semantic Query \(Q_s\) are concatenated and transformed via an MLP, followed by attention computation with the semantic Key/Value: \(f_s = \text{softmax}(\frac{\mathrm{Mf_1}(\text{Cat}(Q_b, Q_s)) K_s^T}{\sqrt{d_k}}) V_s\)
   • Design Motivation: Boundary and semantic information play different roles in the attention mechanism—the boundary Query carries "where is the boundary" information. Fusing it with the semantic Query allows the model to simultaneously consider semantic similarity and boundary awareness during attention computation. Simply concatenating boundary and semantic features (as in existing methods) is less effective than fusion at the Query level.

2. Multi-Layer Feature Extraction:

   • Function: Extracts multi-scale features containing global and local information.
   • Mechanism: Leverages octree levels 8-11 features, utilizing upsampling and 1×1/3×3 convolutions for cross-layer interaction, fusing feature information from different depths to form \(f_o\).
   • Design Motivation: Parent nodes contain global information while child nodes preserve local details. Multi-layer interaction captures both simultaneously.

3. Parallel Boundary Pseudo-Label Calculation (PBPLC):

   • Function: Real-time calculation of boundary pseudo-labels during training to support data augmentation.
   • Mechanism: Powered by CUDA parallel computing, each point is treated as a center point, checking if neighbors within radius \(r\) possess different semantic labels. Handled in parallel with CUDA threads, the complexity is reduced from \(\mathcal{O}(n^2)\) to \(\mathcal{O}(n)\).
   • Design Motivation: Existing methods (such as CBL) require offline pre-processing to compute boundary labels, which is incompatible with data augmentations like mixup. The real-time computation method is 3.9× faster than CBL (46.3ms vs 179.2ms) and is naturally compatible with all data augmentations.

Loss & Training

• Semantic segmentation loss: \(\mathcal{L}_{sem} = \text{CE} + \text{Dice Loss}\)
• Boundary segmentation loss: \(\mathcal{L}_{bou} = \text{BCE} + \text{Dice Loss}\)
• Training setup: 4× RTX 4090, 400 epochs, Adam optimizer, learning rate 0.001 with cosine annealing, boundary radius of 6cm.
• Test-time augmentation (TTA): Rotation + superpoint pooling + checkpoint ensemble.

Key Experimental Results

Main Results

Dataset	Metric	BFANet	OctFormer(Baseline)	PTv3	Gain (vs Baseline)
ScanNet200 Test	mIoU	36.0	32.6	39.2(+PPT auxiliary data)	+3.4
ScanNet200 Test	Head	55.3	53.9	59.2	+1.4
ScanNet200 Test	Common	29.3	26.5	33.0	+2.8
ScanNet200 Test	Tail	19.3	13.1	21.6	+6.2
ScanNet200 Val	mIoU	37.3	32.6	35.2	+4.7
ScanNetv2 Val	mIoU	78.0	75.7	77.5	+2.3

Ablation Study

Configuration	mIoU↑	FErr↓	MErr↓	RErr50↓	DErr50↓
Baseline (OctFormer)	32.7	33.7	37.7	20.2	20.1
+Boundary Prediction	33.7	32.6	36.4	20.0	19.9
+B-S Block	36.4	30.1	34.7	18.6	18.7
+B-S Block +TTA	37.3	31.3	35.9	18.1	18.6

Key Findings

• Compared to simple boundary prediction, the B-S Block brings an additional 2.7% mIoU improvement, demonstrating that Query fusion is superior to feature concatenation.
• The most significant improvements are observed in boundary analysis metrics: FErr -3.6%, MErr -3.0%, DErr -1.4% (vs baseline).
• The largest gains are achieved on Common and Tail categories (mainly small objects): Common +4.9%, Tail +4.7% (vs baseline).
• PBPLC takes only 46.3ms, which is 3.9× faster than CBL.
• Ranked 2nd on the ScanNet200 leaderboard (the highest score without utilizing auxiliary data).
• TTA slightly degrades FErr and MErr (maxpooling ensemble is prone to extreme values for small regions).

Highlights & Insights

• The paradigm of examining segmentation problems from the perspective of "error classification" is highly valuable—not only does it propose a method, but it also introduces new evaluation dimensions.
• The four types of error metrics (RErr/DErr/MErr/FErr) offer long-term contribution value to the segmentation community.
• The design of selectively fusing boundary and semantic information within Q/K/V of the attention mechanism is elegant—distinguishing itself from simple feature concatenation.
• Real-time CUDA boundary label computation constitutes a practical engineering contribution.

Limitations & Future Work

• The proposed method mainly improves boundary-related errors, with limited improvement on region classification errors (RErr).
• Currently validated only in indoor scenes, it could be extended to outdoor autonomous driving, as well as urban/forest point clouds.
• Maxpooling ensemble in TTA negatively impacts small region segmentation, and better ensemble strategies could be explored.
• The boundary radius \(r\) is fixed (6cm); adaptive radius selection could be considered.

Related Work & Insights

• Relationship with OctFormer: BFANet adds a boundary analysis module on top of OctFormer, boosting mIoU by 3.4%.
• Relationship with PTv3: Under the setting without auxiliary data, it outperforms PTv3 on the validation set (37.3 vs 35.2). However, PTv3+PPT (large-scale pre-training) remains stronger.
• Relationship with CBL/JSENet: While all leverage boundary information, BFANet integrates it at the Q/K/V level rather than simple concatenation and proposes a faster pseudo-label computation.
• Insight: Fine-grained error analysis is an important tool for understanding and improving segmentation models.

Supplementary Analysis

Definition and Recommendations for Four Error Metrics

• RErr\(_\theta\): Region classification error rate, measuring the proportion of correctly classified regions among those with IoU > \(\theta\).
• DErr\(_\theta\): Displacement error, measuring the alignment accuracy of boundary region points (excluding interference from merge and false response).
• FErr: False response rate, measuring the proportion of predicted boundaries that do not belong to the ground-truth (GT) boundaries.
• MErr: Merge error rate, measuring the proportion of GT boundaries not covered by predicted boundaries.
• It is recommended to routinely report these four metrics in 3D segmentation papers to help the community better understand the strengths and weaknesses of different methods.

Model Parameters and Inference Efficiency

• The model has 44.3M parameters for inference.
• Single-scene inference (without TTA) takes 60.7ms for approximately 158.8K points.
• Online boundary pseudo-label computation takes 46.3ms, which does not affect inference speed (only required during training and evaluation).

Rating

• Novelty: ⭐⭐⭐⭐ The four-type error classification and boundary-semantic fusion at the Q/K/V level are novel, though the overall framework is built upon established methods.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Dual evaluation with both traditional and new metrics, comprehensive ablation studies, including official test set submission results.
• Writing Quality: ⭐⭐⭐⭐⭐ Clearly defined problems, intuitive visualization of error classification, and rigorous logic throughout the paper.
• Value: ⭐⭐⭐⭐ The four types of error metrics hold long-term value for the community, and the method achieves SOTA performance without utilizing auxiliary data.

Related Papers

• [CVPR 2025] 3D Dental Model Segmentation with Geometrical Boundary Preserving
• [CVPR 2025] COB-GS: Clear Object Boundaries in 3DGS Segmentation Based on Boundary-Adaptive Gaussian Splitting
• [CVPR 2025] Rewis3d: Reconstruction Improves Weakly-Supervised Semantic Segmentation
• [CVPR 2025] JOPP-3D: Joint Open Vocabulary Semantic Segmentation on Point Clouds and Panoramas
• [CVPR 2026] Image-to-Point Cloud Feature Back-Projection for Multimodal Training of 3D Semantic Segmentation