G2P: Gaussian-to-Point Attribute Alignment for Boundary-Aware 3D Segmentation

Conference: ECCV 2026
arXiv: 2601.03510
Code: To be confirmed（project page: https://hojunking.github.io/webpages/G2P/）
Area: 3D Vision
Keywords: 3D point cloud segmentation, 3D Gaussian Splatting, geometric bias, boundary awareness, knowledge distillation

TL;DR

The paper proposes G2P, which establishes precise Gaussian-to-point correspondences using Mahalanobis distance and transfers opacity and scale attributes from 3D Gaussian Splatting to point clouds. This introduces appearance cues while preserving the original point geometry, effectively reducing segmentation confusion between objects with similar geometry but different appearances.

Background & Motivation

Point cloud semantic segmentation is a core task in three-dimensional scene understanding. Although architectures such as MinkUNet, OctFormer, and Point Transformer v3 continue to improve segmentation accuracy, the inherent sparsity and irregular sampling of point clouds force models to rely excessively on coarse geometric cues, creating what is termed geometric bias. In indoor scenes, objects such as doors, windows, and refrigerators are often coplanar with or adjacent to walls. Point clouds provide nearly identical geometric information for these regions, without sufficient appearance evidence such as color, texture, and material to distinguish them. For example, a refrigerator's reflective surface and a white wall are almost geometrically indistinguishable, and purely geometric models often misclassify them as the same plane.

Existing work addresses this problem from two directions. Boundary-aware methods such as JSENet, CBL, and BFANet explicitly learn edge features to refine segmentation boundaries, but remain limited to geometric reasoning and cannot introduce appearance information to distinguish geometrically similar objects with different appearances. 2D-3D fusion methods such as VMVF and ODIN attempt to inject rich visual features from RGB images, but structural misalignment between discrete point distributions and 2D projections makes thin structures difficult to distinguish from planes. Occluded regions also suffer from lost projected information and point-to-pixel alignment errors. These fundamental structural mismatches make it difficult for 2D-3D fusion to truly unify geometry and appearance in native 3D space.

3D Gaussian Splatting (GS) offers a new perspective. GS represents a scene as a collection of anisotropic Gaussian primitives, each carrying continuous volumetric geometry and appearance attributes and sharing the same 3D coordinate system as the point cloud. However, adaptive density control during GS optimization—pruning, cloning, and splitting—changes the distribution of Gaussian positions, causing the primitives to deviate from the original point cloud geometry. Directly using Gaussian coordinates for segmentation can therefore introduce geometric distortion and blurred boundaries. The paper's core insight is to retain the precise geometry of the original point cloud as input and use only GS-derived opacity and scale as auxiliary cues, injecting them into point representations through carefully designed Gaussian-to-point correspondences to introduce appearance information and boundary cues without compromising geometric accuracy.

Method

Overall Architecture

G2P has two stages. The first, the preparation stage, comprises three steps: (1) Gaussian-to-point feature alignment searches within a radius for the k nearest Gaussian primitives to each point using Mahalanobis distance, then aggregates their opacity and scale attributes with anisotropic covariance-aware inverse-distance weighting to obtain enhanced point representations; (2) Gaussian-scale-based boundary pseudo-label extraction uses the aggregated scale distribution—small scales concentrate at object boundaries, while large scales cover flat regions—to identify scale-based boundary candidates, then takes their union with semantic boundaries to generate boundary pseudo-labels; (3) appearance encoder pretraining trains a Sonata-based appearance encoder from scratch on enhanced points comprising coordinates+color+opacity. The second stage is training: a PT v3 backbone jointly predicts semantic segmentation and boundary maps, with the boundary head supervised by pseudo-labels. Meanwhile, the appearance encoder acts as a teacher, transferring appearance feature knowledge to the student backbone through a cosine-similarity distillation loss. Semantic, boundary, and distillation losses jointly optimize the model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    P["Original point cloud<br/>(coordinates+color+normals)"] --> A["Gaussian-to-point feature alignment<br/>Mahalanobis matching→aggregation<br/>opacity+scale"]
    A --> B["Enhanced point cloud<br/>(coordinates+color+normals+<br/>opacity+scale)"]
    B --> C["Appearance encoder pretraining<br/>Sonata architecture·opacity as input"]
    B --> D["Scale-based boundary extraction<br/>small scales→boundary candidates<br/>∪semantic boundaries→pseudo-labels"]
    C --> E["Appearance distillation<br/>teacher→backbone·cosine similarity"]
    D --> F["Segmentation backbone training<br/>PT v3 + B-S block<br/>semantic head+boundary head"]
    E --> F
    F --> G["Final segmentation prediction<br/>Gaussian-free inference"]

Key Designs

1. Covariance-aware Gaussian-to-point feature alignment

After adaptive density optimization, Gaussian primitives deviate from their initial positions, so directly using their coordinates introduces geometric distortion and boundary blur. Transferring their appearance attributes to the original points while preserving point geometry would instead combine the strengths of both representations. G2P does this by taking each point \(p_i\) and first filtering candidate Gaussians within a Euclidean radius \(r^g\), then selecting the k nearest neighbors from these candidates using Mahalanobis distance. The key is the anisotropic information in the Gaussian covariance matrix \(\Sigma\): it encodes Gaussian rotation and scale in the distance metric, giving distance along the long axis a different physical meaning from distance along the short axis. This better captures proximity in real three-dimensional space than isotropic Euclidean distance. For the selected k neighbors, normalized inverse Mahalanobis distances serve as weights to aggregate their scales \(S_j^g\) and opacities \(\alpha_j^g\):

\[S_i^p = \sum_{j=1}^{k} w_{ij} \cdot S_j^g, \quad \alpha_i^p = \sum_{j=1}^{k} w_{ij} \cdot \alpha_j^g\]

After aggregation, each point expands from 9 dimensions (coordinates+color+normals) to 13 dimensions (coordinates+color+normals+scale+opacity). Ablations show that Mahalanobis distance outperforms Euclidean distance and that \(k=20\) is the optimal neighborhood size.

2. Gaussian-scale-based boundary pseudo-label extraction

Object boundaries in indoor scenes often correspond to geometric discontinuities, but these discontinuities are weak for coplanar objects such as built-in refrigerators against walls or windows in walls. The authors find that Gaussian scale offers a natural geometric signal: during anisotropic Gaussian optimization, small Gaussians densely populate object boundaries and thin structures to preserve rendering accuracy, while large Gaussians dominate flat regions. Based on this observation, G2P first excludes background classes such as walls and floors from the enhanced point cloud, then computes the L2 norm \(\|S_i'\|_2\) of each point's scale vector. It discards the fraction \(\eta\) of points with the largest norms, effectively retaining small-scale regions, to obtain the scale-based boundary candidate set \(\mathcal{B}_{\text{scale}}\). Scale alone may be noisy because of texture or photometric variation, so semantic boundaries \(\mathcal{B}_{\text{sem}}\) are additionally derived from semantic labels: points whose local radius \(r^s\) contains a different class label. The union of the two sets forms the final boundary pseudo-labels. This requires no manually annotated boundaries, and the unsupervised nature of scale makes it naturally applicable to unlabeled scenes.

3. Opacity-guided appearance distillation

In the training stage, G2P no longer depends on Gaussian primitives, instead using only point clouds as input for inference—a "Gaussian-free" design. To achieve this, G2P first trains a Sonata-based appearance encoder from scratch during preparation, using enhanced point representations \(( \mu^p, c, \alpha' )\) with normals replaced by opacity. Here, opacity acts as a "view-consistent appearance confidence signal": background regions such as walls and floors remain visible across views and therefore have uniformly high opacity, while foreground objects have diverse opacity distributions due to geometric complexity and self-occlusion. During training, the appearance encoder is frozen as the teacher. Features extracted by the PT v3 backbone pass through an MLP mapping layer \(\phi\), then a cosine-similarity loss compares them with the teacher features:

\[\mathcal{L}_{\text{distill}} = \frac{1}{N} \sum_{i=1}^N \left(1 - \frac{\phi(f_i^p) \cdot f_i^a}{\|\phi(f_i^p)\|_2 \|f_i^a\|_2}\right)\]

This distillation loss is optimized jointly with the semantic segmentation loss (cross-entropy + Lovász-softmax) and boundary supervision loss (BCE + Dice). Ablations show that distillation outperforms directly feeding enhanced features into the backbone or fine-tuning the appearance encoder. Distillation allows the backbone to learn appearance discrimination patterns rather than merely depend on additional input channels.

Loss & Training

The overall loss is a weighted sum of three terms: \(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{sem}} + \lambda_b \mathcal{L}_{\text{bou}} + \lambda_d \mathcal{L}_{\text{distill}}\), with \(\lambda_d=0.4\) and \(\lambda_b=0.9\). The backbone is trained on a single RTX 3090 for 800 epochs with a batch size of 4 and AdamW at an initial learning rate of 0.003. The appearance encoder is pretrained separately for each dataset on an A6000 for 400 epochs. Gaussian alignment uses a search radius of \(r^g=0.06\)m, \(k=20\) neighbors, a scale trimming ratio of \(\eta=0.7\), and a semantic boundary radius of \(r^s=0.04\)m.

Key Experimental Results

Main Results

Dataset	Metric	PT v3 (baseline)	BFANet	UniPre3D	G2P (Ours)	Gain
ScanNet v2 (mIoU)	20 classes	77.5	78.0	77.6	78.4	+0.9 vs PT v3
ScanNet200 (mIoU)	200 classes	35.2	—	36.0	36.6	+1.4 vs PT v3
ScanNet200 (OA)	200 classes	83.6	—	83.7	83.8	+0.2
ScanNet++ (mIoU)	100 classes	47.9	—	—	48.7	+0.8
Matterport3D (mIoU)	21 classes	55.5	—	—	55.9	+0.4

Method	Geometrically distinguishable classes (Avg)	Geometrically challenging classes (Avg)	Refrigerator IoU	Shower curtain IoU
PT v3	84.5	66.8	64.9	68.9
BFANet	84.7	67.3	69.3	70.8
G2P	85.8	69.2	70.9	76.1

Ablation Study

Config	mIoU	Note
PT v3 baseline	77.0	Purely geometric baseline
+ Boundary supervision (Boundary only)	77.8	Adds only GS-scale boundary pseudo-labels
+ Appearance distillation (Distillation only)	77.8	Adds only GS-opacity distillation
G2P (Full)	78.4	Combining both performs best

Alignment metric	mIoU	Note
Euclidean distance	77.0	Ignores anisotropy
Mahalanobis distance (k=10)	77.4	Too few neighbors
Mahalanobis distance (k=20)	78.4	Best trade-off
Mahalanobis distance (k=30)	77.9	Too many neighbors cause oversmoothing

Key Findings

• The largest gains occur in two geometrically challenging classes: refrigerator IoU exceeds PT v3 by +6.0, and shower curtain IoU by +7.2, indicating that GS appearance cues help distinguish objects with similar geometry but different appearances
• Boundary supervision and appearance distillation each independently improve mIoU by +0.8 and complement each other well; increasing the distillation weight \(\lambda_d\) improves appearance-challenging classes but slightly degrades geometric classes, with \(0.4\) providing a good balance
• At inference time, G2P adds only 0.2M parameters (+0.4%), with +21% latency and +74% memory usage; training incurs greater overhead (+67% latency, +30% memory), but this is acceptable in practical applications because inference does not require GS
• G2P performs best on ScanNet200 Tail classes (20.2 mIoU) and also improves instance segmentation (mAP25 +1.7)

Highlights & Insights

• Use GS attributes rather than GS geometry: previous methods that directly segment Gaussian primitives are often constrained by geometric drift after Gaussian optimization. G2P uses GS opacity and scale as auxiliary attributes rather than replacement inputs, preserving the geometric accuracy of the original point cloud.
• Mahalanobis distance instead of Euclidean distance: a simple but effective technique—Gaussians are anisotropic ellipsoids, and Euclidean neighbor search ignores their shape. Mahalanobis distance incorporates the covariance matrix into the metric, matching proximity more faithfully in real scenes; the paper's ablation confirms a +1.4 mIoU gap.
• Scale as an unsupervised boundary signal: during GS optimization, small Gaussians naturally concentrate near object boundaries to render them faithfully. The paper directly reuses this behavior for boundary extraction without manual annotation, complementing semantic boundaries.
• Opacity distillation instead of 2D-3D fusion: unlike complex pipelines such as ODIN that require 2D projection, G2P injects and distills appearance cues in native 3D space, entirely avoiding projection misalignment.

Limitations & Future Work

• The current method depends on an offline preparation stage comprising GS reconstruction + appearance encoder pretraining, limiting extension to outdoor scenes. Dynamic lighting, large scale variations, and sparse viewpoints make outdoor GS reconstruction unstable, requiring locally normalized scale statistics or more robust GS methods.
• Training incurs substantial additional overhead, with memory usage of 7.3G versus PT v3's 5.6G, making it less suitable for large-scale scenes requiring rapid iteration. However, GS can be entirely removed at inference time, making the approach suitable for offline mapping.
• All evaluated indoor datasets use GS reconstructions provided by SceneSplat-7K; adaptability to outdoor and urban scenes such as SemanticKITTI and nuScenes remains to be verified.
• Potential improvements include combining lightweight GS methods such as DNGaussian and Speedy-Splat to reduce preparation costs, and extending the approach to multimodal settings or open-vocabulary semantic segmentation.

Related Work & Insights

• vs BFANet / JSENet: these methods also target boundary-aware segmentation, but detect boundaries solely from geometric discontinuities. G2P uses GS scale signals to supply additional appearance-level information, offering clear advantages for coplanar objects.
• vs ODIN / VMVF: these 2D-3D fusion methods require 2D feature projection and suffer from projection misalignment and occlusion-related information loss. G2P operates entirely in 3D space without projection, yielding a simpler structure.
• vs UniPre3D: UniPre3D also uses GS for pretraining, but relies on rendering losses as self-supervision rather than directly transferring attributes. G2P learns through explicit Gaussian-to-point alignment under 3D supervision, providing more direct semantics.
• vs segmentation directly using GS coordinates: the ablation shows that directly using Gaussian coordinate points (\(\mu^g, c', n', \alpha\)) achieves only 73.1 mIoU, far below the PT v3 baseline. Positional drift of Gaussian primitives thus severely harms segmentation quality, indirectly demonstrating the need for G2P's strategy of preserving point geometry while attaching GS attributes.

Rating

• Novelty: ⭐⭐⭐⭐ A novel approach to transferring GS attributes into point cloud segmentation; Mahalanobis alignment and scale-based boundary extraction are straightforward but effective
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Validated on four datasets—ScanNet v2/200/++ and Matterport3D—with detailed ablations covering boundary sources, alignment metrics, loss weights, and class-group analysis
• Writing Quality: ⭐⭐⭐⭐ Clear methodological logic; figures, tables, and class-group analysis make the sources of improvement intuitive, while some ablation details appear in the supplementary material
• Value: ⭐⭐⭐⭐ A directly reusable feature enhancement approach that leaves mainstream segmentation inference pipelines unchanged, with no GS dependency at inference time, making it highly practical

Related Papers

• [CVPR 2026] Geometry-Aware Cross-Modal Graph Alignment for Referring Segmentation in 3D Gaussian Splatting
• [ECCV 2026] TORA: Topological Representation Alignment for 3D Shape Assembly
• [CVPR 2026] Tavatar: Topology-Aware Gaussian Attribute Derivation for Animatable Human Avatars
• [ECCV 2026] BrepLLM: Enabling Large Language Models to Understand Boundary Representations
• [CVPR 2025] 3D Dental Model Segmentation with Geometrical Boundary Preserving