Multi-Modal Assistance for Unsupervised Domain Adaptation on Point Cloud 3D Object Detection

Conference: AAAI 2026
arXiv: 2511.07966
Code: github.com/liangp/MMAssist
Area: 3D Vision
Keywords: Unsupervised Domain Adaptation, 3D Object Detection, Multi-Modal Fusion, Point Cloud, Pseudo-Labeling

TL;DR

This paper proposes MMAssist, which leverages image and text features as a "bridge" to align 3D features between the source and target domains, while combining 2D detection results to enhance the quality of pseudo-labels, significantly improving the performance of LiDAR-based 3D unsupervised domain adaptation for object detection.

Background & Motivation

LiDAR-based 3D object detection is crucial in autonomous driving. However, due to factors such as differences in LiDAR beam numbers and environmental variations across different domains (datasets), the performance of a trained model often drops significantly when transferred to a new domain. Existing 3D unsupervised domain adaptation (3D UDA) methods mainly rely on teacher-student self-training frameworks and pseudo-labels, but most of these methods only use point cloud data, ignoring the simultaneously captured image information.

Key Insight:

Smaller domain gap in image features: Representations learned by pre-trained vision models on large-scale data possess strong generalization capabilities. The gap between image features of similar objects in different domains is much smaller than that between point cloud features.

Cross-domain consistency of text descriptions generated by LVLMs: Large vision-language models (such as LLaVA) generate highly similar text descriptions for similar objects in different domains (e.g., cars in Waymo and nuScenes are described with similar text).

Poor quality of raw pseudo-labels for distant objects: 3D detectors have limited detection capabilities in distant regions, while 2D detectors can still detect these objects relatively well in images.

Therefore, image and text features can serve as a "bridge" to indirectly align 3D features of the two domains, and 2D detection results can supplement the lack of distant pseudo-labels.

Method

Overall Architecture

MMAssist is based on a teacher-student self-training framework (DTS) and consists of two phases: - Pre-training phase: Train the source domain model on labeled source data, while performing 3D-image-text feature alignment. - Self-training phase: Initialize the teacher and student models with the source domain model. The student is trained on the target domain using pseudo-labels, while the teacher is updated via EMA.

Key characteristic: Image and text information are only utilized during the training phase. Inference requires only point cloud input, introducing no additional computational overhead during inference.

Key Designs

1. Cross-Domain Feature Alignment (Bridged by Image/Text)

For each 3D bounding box (ground truth or pseudo-label), it is projected onto the 2D image plane using camera intrinsic and extrinsic parameters to obtain a 2D box. Then: - Image features: Use RoIAlign to extract image features \(\mathbf{f}_i^{img} \in \mathbb{R}^{C^{img}}\) of the 2D box from a pre-trained GroundingDINO backbone. - Text features: Use LLaVA to generate text descriptions of the target ("There is a {class} in the area ..., please describe the characteristics"), and then use the SLIP text encoder to extract text features \(\mathbf{f}_i^{text} \in \mathbb{R}^{C^{text}}\).

For the 3D boxes predicted by the detector, their 3D features are extracted and mapped to the image and text spaces via MLPs to align with the corresponding image/text features.

Image alignment loss (contrastive learning style, pulling positive samples closer and pushing background farther): \(\mathcal{L}_{align}^{img} = \frac{1}{L'}\sum_{i=1}^{L'}\max\left(\frac{1}{N^{bg}}\sum_{j=1}^{N^{bg}}\text{sim}(\hat{\mathbf{f}}_i^{img}, \mathbf{g}_j^{bg}) - \text{sim}(\hat{\mathbf{f}}_i^{img}, \hat{\mathbf{g}}_i^{img}) + \sigma, 0\right)\)

Text alignment loss (cosine similarity): \(\mathcal{L}_{align}^{text} = \frac{1}{L'}\sum_{i=1}^{L'}\left(1 - \text{sim}(\hat{\mathbf{f}}_i^{text}, \hat{\mathbf{g}}_i^{text})\right)\)

Design Motivation: 3D-image/text alignment is performed in both the source and target domains respectively. Due to the high cross-domain consistency of image and text features, 3D features are indirectly pulled closer, achieving implicit cross-domain alignment.

1. Multi-Modal Feature Fusion

3D features, image-aligned features, and text-aligned features are fused for final prediction: - First, use MLPs to unify the dimensions of the three. - Concatenate the three features and feed them into another MLP to learn a weight vector \(\mathbf{w} \in \mathbb{R}^3\). - Weighted fusion: \(\mathbf{f}^{fused} = \mathbf{w}_0 \mathbf{f}^{3D} + \mathbf{w}_1 \mathbf{f}^{img} + \mathbf{w}_2 \mathbf{f}^{text}\)

The fused features are used for the second-stage refinement of PV-RCNN, refinement of PointPillars, and IoU prediction of SECOND-IoU.

1. Student-Teacher 3D Feature Alignment

During the self-training phase, the 3D features of the matched boxes predicted by the student and teacher models are additionally aligned: \(\mathcal{L}_{ST} = \frac{1}{G}\sum_{i=1}^{G}\left(1 - \text{sim}(\hat{\mathbf{f}}_i^S, \hat{\mathbf{f}}_i^T)\right)\)

1. Pseudo-Label Enhancement Based on 2D Detection

GroundingDINO is used to detect 2D boxes on target domain images, which are then lifted to 3D space through geometric reasoning. New pseudo-labels are filtered based on two conditions: - Distance condition: Only retain distant 3D boxes with a distance \(\ge \tau\) (30m). - Overlap condition: The IoU with teacher pseudo-labels must be \(\le \xi\) (0.5) to avoid duplication.

Final pseudo-labels = teacher pseudo-labels \(\cup\) new image pseudo-labels.

Loss & Training

Pre-training phase: \(\mathcal{L}_{pre} = \mathcal{L}_{det} + \alpha \mathcal{L}_{align}^{text} + \beta \mathcal{L}_{align}^{img}\)

Self-training phase: \(\mathcal{L}_{student} = \mathcal{L}_{det} + \alpha \mathcal{L}_{align}^{text} + \beta \mathcal{L}_{align}^{img} + \gamma \mathcal{L}_{ST}\)

• \(\alpha = \beta = 0.3\) (pre-training) / \(0.03\) (self-training), \(\gamma = 0.1\)
• EMA coefficient \(\epsilon = 0.999\)
• Self-training for \(30\) epochs, learning rate of \(1.5\times10^{-3}\)
• IoU matching thresholds \(\mu = \eta = \xi = 0.5\)

Key Experimental Results

Main Results

Domain adaptation is performed among three datasets (Waymo, nuScenes, KITTI) using three detectors (PV-RCNN, PointPillars, SECOND-IoU). The proposed method achieves the best performance in 7 out of 9 sub-tasks:

Task	Detector	Ours AP_BEV/AP_3D	Prev. SOTA AP_BEV/AP_3D	Gain
N→K	PV-RCNN	86.8/78.1	85.8/75.5 (CMT)	+1.0/+2.6
W→K	PV-RCNN	87.6/72.7	85.9/74.5 (CMT)	+1.7/-1.8
W→N	PV-RCNN	45.5/27.0	44.4/26.4 (CMDA)	+1.1/+0.6
N→K	PointPillars	81.9/60.4	81.9/52.8 (GroupEXP-DA)	0/+7.6
W→K	PointPillars	81.4/56.8	78.4/54.1 (GroupEXP-DA)	+3.0/+2.7
N→K	SECOND-IoU	84.8/69.8	83.0/68.1 (CMT)	+1.8/+1.7

Inference speed is almost unaffected: SECOND-IoU 52.04 \(\rightarrow\) 51.40 FPS, PV-RCNN 6.67 \(\rightarrow\) 6.65 FPS, PointPillars 82.55 \(\rightarrow\) 79.43 FPS.

Ablation Study

Configuration	AP_BEV	AP_3D	Description
(a) Baseline (DTS)	76.7	52.7	Baseline
(b) + Image Pseudo-labels	79.1	53.2	Improved performance on distant detections
(c) + Image Alignment + STA	80.5	54.4	Image bridge effectiveness
(d) + Text Alignment + STA	80.0	54.5	Text bridge effectiveness
(e) + Image & Text Alignment	80.9	55.7	Complementarity of dual bridges
(f) Full MMAssist	81.4	56.8	Synergy of all components

Verification of the bridge effect: incorporating alignments in both pre-training and self-training phases (81.4/56.8) is far superior to doing so in only one of the phases (~79.7/55.3), validating the efficacy of the "bridge".

Key Findings

• Image pseudo-labels mainly improve AP_3D by \(+3.3\%\) in the 30-60m range, and by \(+0.5\%\) in the 60-150m range.
• Weighted fusion (WSum) outperforms concatenation (Concat: 80.5/55.4) and direct summation (Sum: 78.8/50.8).
• The text feature generation scheme of LLaVA + SLIP performs better than Qwen2-VL + SLIP and LLaVA + LLaMA.

Highlights & Insights

1. Novel bridge concept: Instead of directly aligning 3D features between two domains (which is difficult), it indirectly achieves cross-domain alignment through image/text features, providing an elegant indirect alignment strategy.
2. No extra overhead at inference: Multi-modal information is used only during training. Inference requires only point clouds, making it highly practical.
3. Strong compatibility: Effectively integrated into three mainstream detectors, demonstrating good generalizability.
4. Simple yet effective supplementation of distant pseudo-labels: Leverages 2D detectors to compensate for the blind spots of 3D detectors at long distances.

Limitations & Future Work

• Alignment is only performed at the instance level, lacking the utilization of global semantic information.
• Dependence on LVLMs (LLaVA) to generate text descriptions increases the complexity of the training pipeline.
• Only verified on the "Car" category, requiring further validation for multi-class generalizability.
• When camera configurations vary significantly across different domains, the quality of 2D projection might be limited.

Related Work & Insights

• DTS (CVPR 2023): The baseline method of this paper, employing random beam re-sampling + teacher-student framework.
• CMT (ACM MM 2024): Hybrid domain alignment, which this paper surpasses under most configurations.
• CMDA (AAAI 2024): Utilizes image semantic knowledge to assist source-domain training, but does not apply it directly to target-domain self-training.
• Insight: Utilizing the cross-domain generalization capability of pre-trained foundation models to assist domain adaptation for specific tasks can be extended to other perception tasks.

Rating

• Novelty: ⭐⭐⭐⭐ (The bridge alignment concept is novel, but the overall framework is still teacher-student-based)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (3 datasets \(\times\) 3 detectors, comprehensive ablation studies)
• Writing Quality: ⭐⭐⭐⭐ (Clear and easy to understand with complete mathematical formulations)
• Value: ⭐⭐⭐⭐ (Highly practical with no additional inference overhead)

Related Papers

• [CVPR 2026] CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptation
• [ECCV 2024] Progressive Classifier and Feature Extractor Adaptation for Unsupervised Domain Adaptation on Point Clouds
• [CVPR 2026] QD-PCQA: Quality-Aware Domain Adaptation for Point Cloud Quality Assessment
• [AAAI 2026] DAPointMamba: Domain Adaptive Point Mamba for Point Cloud Completion
• [AAAI 2026] Distilling Future Temporal Knowledge with Masked Feature Reconstruction for 3D Object Detection