Sonata: Self-Supervised Learning of Reliable Point Representations

Conference: CVPR 2025
arXiv: 2503.16429
Code: GitHub
Area: 3D Vision
Keywords: Point cloud self-supervised learning, geometric shortcut, linear probe, self-distillation, semantic representation

TL;DR

Proposes Sonata, a reliable point cloud self-supervised learning method. By identifying and resolving the "geometric shortcut" issue (where representation collapses into low-level spatial features such as surface normals or point height), it increases the linear probe mIoU on ScanNet from 21.8% to 72.5% (a 3.3x improvement) and achieves SOTA on multiple 3D perception tasks.

Background & Motivation

• Image self-supervised learning is highly mature; linear probing can closely match fine-tuning performance, and the semantic nature of representations can be directly visualized via PCA.
• Point cloud self-supervised learning is still in its infancy, and 3D SSL methods are rarely integrated into practical application pipelines such as autonomous driving and robotics.
• Existing 3D SSL methods achieve only 21.8% mIoU on ScanNet linear probing, which is far below the 63.1% achieved by DINOv2 3D-aggregated features.
• The authors identify "geometric shortcuts" as the root cause: models collapse into easily accessible low-level geometric cues (such as normal direction and point height).
• This issue is unique to 3D: the sparsity of point clouds makes it inevitable that point coordinate information is introduced into operators (unlike images, where all information resides within the input features).
• The U-Net decoder enforces point-wise features at the original resolution, introducing local geometric cues and exacerbating geometric shortcuts.
• The SSL loss of existing methods decreases rapidly during the early stages of training, showing that the model "does not struggle enough," which indicates the presence of a shortcut.
• The lack of a reliable point cloud foundation representation model seriously hinders the development of the 3D domain.

Method

Overall Architecture

Sonata adopts a point cloud self-distillation framework. Based on Point Transformer V3 (PTv3, 108M parameters), it is trained for 200 epochs on 140k scene-level point clouds. Local views (5% to 40% sampling) and global views (40% to 100% sampling) are generated, along with a masked view (70% masking) based on the global view. The student model encodes the local and masked views, while the teacher model (updated via EMA) encodes the global views. Feature embeddings of corresponding points are aligned through spatial matching. A self-distillation criterion combining Sinkhorn-Knopp centering, KoLeo regularization, and clustering assignment is employed.

Key Designs

Design 1: Decoder Removal + Feature Up-casting - Function: Fundamentally reduces the impact of geometric shortcuts while retaining multi-scale features. - Mechanism: Removes the U-Net decoder and performs self-distillation only on the encoder output. Hierarchical pooling naturally disrupts the position information of point coordinates, and the feature channels are increased from 96 to 512. To compensate for the loss of multi-scale information, non-parametric feature up-casting (similar to hypercolumns) is introduced, up-casting features stage-by-stage back to the resolution of previous encoding stages and concatenating them. - Design Motivation: U-Net decoding at the original resolution inevitably introduces local geometric cues; removing the decoder leads to a massive leap in linear probing from 20.7% to 60.4%, making it the most critical design.

Design 2: Masked Point Jittering + Progressive Parameter Scheduling - Function: Further suppresses the model's reliance on spatial coordinate information. - Mechanism: Applies stronger Gaussian jittering (\(\sigma=0.01\) vs standard \(\sigma=0.005\)) to masked points to disrupt their spatial relationships. A progressive scheduling strategy is adopted: the mask size increases gradually from 10cm to 40cm, the mask ratio rises from 30% to 70%, the teacher temperature increases from 0.04 to 0.07, and the weight decay scales from 0.04 to 0.2. - Design Motivation: Models are most prone to degrade to spatial cues when masked points lack input features; progressive difficulty scaling encourages the model to learn from input features first before adapting to harder tasks, similar to curriculum learning.

Design 3: Large-Scale Multi-Dataset Joint Training - Function: Scales the dataset size to enhance the generalization of representations. - Mechanism: Aggregates 140k scenes across 7 data sources (ScanNet, ScanNet++, S3DIS, ArkitScenes, HM3D, Structured3D, ASE), which is 86.7x larger than the data size used in PointContrast. All BN layers in PTv3 are replaced with LN to enhance domain adaptability. - Design Motivation: Unsupervised learning removes the constraints of manual annotation, allowing massive scaling of data; replacing BN with LN avoids domain bias during joint training on multiple datasets.

Loss & Training

The DINOv2-style self-distillation criterion is adopted: Sinkhorn-Knopp centering prevents mode collapse, KoLeo regularization encourages uniform feature distribution, and clustering assignment serves as the supervision signal. Contrastive learning (limited by the number of point pairs) and generative learning (anchored to predefined low-level cues) are not used.

Key Experimental Results

Main Results: ScanNet Semantic Segmentation Linear Probe (mIoU)

Method	Data Size	Linear Probe	Decoder Probe	Fine-tune
PointContrast	1.6k	5.6	-	73.7
MSC	6.7k	21.8	-	77.6
DINOv2 (3D aggregated)	-	63.1	-	-
Sonata	140k	72.5	75.3	79.8
Sonata + DINOv2	-	76.4	-	-

Ablation Study: Design Evolution (ScanNet Linear Probe mIoU)

Design Step	Linear Probe	Gain
Baseline (MSC + PTv3)	20.7	-
+ Self-distillation	23.4	+2.7
+ Decoder Removal	60.4	+37.0
+ Feature Up-casting	63.4	+3.0
+ Masked Point Jittering	65.1	+1.7
+ Progressive Scheduling	67.2	+2.1
+ Data Scaling 140k	72.5	+5.3

Key Findings

• Decoder removal is the most critical design (increasing linear probing by 37 points), verifying the geometric shortcut hypothesis.
• Sonata features outperform DINOv2 3D-aggregated features (72.5% vs 63.1%), indicating that Sonata captures unique 3D information invisible in images.
• Fusing the two further improves performance to 76.4%, demonstrating their complementary nature.
• With only 1% of the data, Sonata improves performance from 25.8% to 45.3%, showing extreme data efficiency.
• Full fine-tuning achieves SOTA across multiple indoor and outdoor 3D perception tasks.

Highlights & Insights

1. Discovery and Resolution of Geometric Shortcuts: Profoundly reveals the fundamental difference between 3D SSL and image SSL—point coordinate information cannot be simply masked out like pixel features.
2. Insight on Decoder Removal: Challenges the conventional U-Net structure in point cloud processing, proving that SSL only requires an encoder.
3. 3.3x Linear Probe Improvement: Marks a milestone toward reliable point cloud SSL.
4. Zero-Shot PCA/K-Means Visualization of Semantic Grouping: Demonstrates for the first time that 3D SSL representations retain high-level semantics.

Limitations & Future Work

• The data scale of 140k scenes is still relatively small compared to image SSL; further scaling could yield greater improvements.
• Training requires 32 GPUs for 200 epochs, making the computational cost relatively high.
• There is still room for improvement in outdoor scenes (e.g., nuScenes).
• The BN to LN replacement incurs an initial accuracy loss in some scenarios.
• Future work can explore deeper integration with 2D foundation models.

Related Work & Insights

• Inspiration is drawn from the history of image SSL (the continuous battle against shortcuts), bringing similar philosophies into 3D.
• Comparison with DINOv2 proves the complementarity of 3D and 2D representations.
• The concept of decoder removal and multi-scale up-casting can be generalized to other 3D tasks that require hierarchical features.

Rating

⭐⭐⭐⭐⭐ — Profound problem identification (geometric shortcuts), simple yet highly effective solutions, and a convincing 3.3x improvement make this a landmark work in the field of 3D self-supervised learning.

Related Papers

• [NeurIPS 2025] Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations
• [CVPR 2026] Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds
• [ICCV 2025] StruMamba3D: Exploring Structural Mamba for Self-supervised Point Cloud Representation Learning
• [ICCV 2025] Towards More Diverse and Challenging Pre-training for Point Cloud Learning: Self-Supervised Cross Reconstruction with Decoupled Views
• [CVPR 2025] SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input