Towards Unsupervised Multi-modal Semantic Segmentation¶
Conference: ECCV 2026
Paper: ECCV Official Poster
Code: https://empactlab.github.io/UMSS/
Area: Segmentation
Keywords: Unsupervised Learning, Multimodal Fusion, Semantic Segmentation, Correspondence Synergy, Modality Conflict Harmonization
TL;DR¶
To tackle the severe fusion degradation where naive multi-modal extensions hurt unsupervised segmentation, UniM2 introduces Cross-modal Correspondence Synergy (CMCS) to discover shared semantic manifolds alongside an asymmetric Cross-modal Harmonizer (CMH) that buffers gradient contradictions, delivering substantial gains of +6.4% mIoU on NYU-Depth-v2 and +9.8% mIoU on MFNet.
Background & Motivation¶
Multi-modal semantic segmentation (MSS) plays an indispensable role in safety-critical applications including autonomous driving, embodied AI, and all-weather robot navigation. Integrating complementary sensing signals such as depth, thermal infrared, or polarization effectively offsets the perceptual blindness of standalone RGB cameras under degraded lighting or adverse weather. However, conventional progress in multi-modal segmentation has been overwhelmingly driven by large-scale, densely labeled datasets. Pixel-level manual annotations are prohibitively expensive and time-consuming to curate, and they inherently constrain models to fixed, closed-world semantic vocabularies, impeding the exploitation of vast uncurated multi-modal sensory streams in the wild.
In parallel, unsupervised semantic segmentation (USS) on unimodal RGB images has advanced rapidly through feature correspondence distillation powered by self-supervised Vision Transformers such as DINO. A straightforward hypothesis is that directly porting established multi-modal fusion schemes—such as element-wise feature addition, convolutional fusion, CBAM attention, or StitchFusion—into unsupervised frameworks like STEGO or EAGLE should bolster segmentation performance. Yet empirical evaluations reveal a striking and counter-intuitive failure: "fusion degradation." Without explicit human supervisory signals to arbitrate heterogeneous sensing modalities, naively incorporating auxiliary depth or thermal features severely compromises clustering quality, resulting in dramatic drops in mIoU. This failure stems from intrinsic cross-modal contradictions. For instance, curtains and walls display sharp textural and color contrasts in RGB imagery but share nearly identical geometric depth planes. In supervised training, ground-truth annotations force models to reconcile these sensor-specific biases; in contrast, unsupervised optimization suffers directly from conflicting relational gradients that shatter the shared manifold's topological structure.
To bridge this critical gap, this paper formalizes the task of Unsupervised Multi-modal Semantic Segmentation (UMSS). Instead of forcing rigid, symmetric alignment across disparate modalities or relying on asymmetric one-way teacher-student distillation, the framework seeks to leverage intrinsic structural agreements across frozen self-supervised representations while softly decoupling physical sensor discrepancies. The core idea is to propose UniM2, a framework that establishes Cross-modal Correspondence Synergy (CMCS) to distill joint cross-sensor agreements into a unified latent space, while employing an asymmetric, RGB-anchored Cross-modal Harmonizer (CMH) to buffer conflicting gradients, consistently turning unsupervised fusion degradation into significant empirical gains.
Method¶
Overall Architecture¶
UniM2 operates on paired multi-modal inputs without human supervision. Taking an RGB-Depth or RGB-Thermal pair as a representative setting, the input images are fed into a frozen self-supervised backbone (DINOv3) to yield high-dimensional dense feature maps \(f_{rgb}\) and \(f_X\). These features pass through lightweight Modality-Specific Networks (MSN) for channel recalibration before being merged via a learnable Conv Fusion module into a unified representation \(f_{fus}\). A compact segmentation head then projects \(f_{fus}\) into low-dimensional semantic manifolds \(s\) for discrete clustering. The framework is trained end-to-end on multi-modal image groups forming Self, KNN, and Random pairs, guided jointly by Cross-modal Correspondence Synergy and Cross-modal Harmonization.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-modal Inputs<br/>RGB Image I + Auxiliary Modality X"] --> B["Frozen DINOv3 Backbone<br/>Extract Dense Features f_rgb & f_X"]
B --> C["Modality-Specific Networks MSN<br/>Channel Recalibration & Refinement"]
C --> D["Learnable Conv Fusion<br/>Aggregate Features into f_fus"]
D --> E["Segmentation Head Projection<br/>Generate Unified Embedding s"]
E --> F["Cross-modal Correspondence Synergy CMCS<br/>Distill Joint Cross-sensor Agreement"]
E --> G["Cross-modal Harmonizer CMH<br/>RGB-anchored Gradient Buffering for f_X"]
F --> H["Joint Unsupervised Optimization<br/>Pixel-level Semantic Clustering"]
G --> H
Key Designs¶
1. Cross-modal Correspondence Synergy (CMCS): Learning Consensus-driven Manifolds Standard unimodal USS assumes that semantic affinities in a low-rank embedding space should mirror the dense correlation matrix of a single feature extractor. In multi-modal environments, however, individual sensors inherently carry idiosyncratic noise and sensor-specific biases. CMCS posits that a reliable semantic space must preserve only structural relationships that are consistently corroborated across constituent sensors. For any training pair \((G_1, G_2)\), feature correlation tensors are computed directly from the frozen backbone as stable anchors: \(F^{rgb}_{hwij} = \cos(f_{rgb,1,hw}, f_{rgb,2,ij})\) and \(F^X_{hwij} = \cos(f_{X,1,hw}, f_{X,2,ij})\). Meanwhile, the unified embedding produces correspondence \(S_{hwij} = \cos(s_{1,hw}, s_{2,ij})\). By distilling correlations from both modalities into \(s\) simultaneously, CMCS rewards feature agreements jointly verified by multiple sensors, effectively filtering out modality-specific clutter without external labels.
2. Cross-modal Harmonizer (CMH): RGB-anchored Asymmetric Buffering Although CMCS emphasizes cross-modal consensus, intrinsic physical sensing discrepancies inevitably introduce conflicting supervisory signals. For example, flat objects with vibrant patterns produce strong semantic boundaries in RGB but appear completely flat and uniform in depth representations. Directly compelling the unified embedding \(s\) to match the auxiliary correlation tensor \(F^X\) backpropagates erroneous smoothing gradients that blur fine-grained RGB boundaries. To resolve this dilemma, UniM2 designs an asymmetric buffering mechanism: RGB is designated as the primary semantic reference and supervises the unified embedding \(s\) directly, whereas auxiliary modality supervision is mediated through a lightweight two-layer convolutional buffer, \(\text{CMH}(\cdot)\). Defining the harmonized embedding as \(s^X = \text{CMH}(s)\), the auxiliary correspondence becomes \(S^X_{hwij} = \cos(s^X_{1,hw}, s^X_{2,ij})\). The auxiliary distillation loss \(\mathcal{L}_X\) is backpropagated through \(s^X\), allowing \(\text{CMH}\) to adaptively regulate the stiffness of the auxiliary constraint. This prevents contradictory geometric gradients from disrupting the primary RGB semantic manifold while still harvesting complementary topological cues.
3. Modular Scalability to Arbitrary Auxiliary Modalities Real-world embodied robots frequently deploy diverse sensor suites beyond simple bi-modal setups. Leveraging the decoupling capability of CMH, UniM2 scales seamlessly to \(N\) auxiliary modalities (e.g., fusing Near-Infrared, Degree of Linear Polarization, and Angle of Linear Polarization simultaneously). Instead of engineering cumbersome multi-branch cross-attention architectures that require delicate hyperparameter balancing, each auxiliary modality is allocated an independent MSN and a dedicated CMH branch. Each auxiliary stream buffers its respective gradient flow independently, preventing cross-sensor gradient tangling and ensuring stable, monotonic gains as auxiliary sensors are progressively incorporated.
Loss & Training¶
The overall training loss of UniM2 is a weighted combination of the primary RGB distillation loss \(\mathcal{L}_{rgb}\) and the harmonized auxiliary losses \(\mathcal{L}_{X_n}\): $\(\mathcal{L}_{cmcs} = \mathcal{L}_{rgb} + \sum_{n=1}^{N} \lambda_n \mathcal{L}_{X_n}\)$ Each component follows the correlation distillation objective with negative pressure: $\(\mathcal{L}_{rgb} = -\sum_{h,w,i,j} (F^{rgb}_{hwij} - b) \max(0, S_{hwij})\)$ $\(\mathcal{L}_{X} = -\sum_{h,w,i,j} (F^{X}_{hwij} - b) \max(0, S^X_{hwij})\)$ where \(b\) denotes a pair-type-dependent scalar bias (Self, KNN, or Random) that penalizes weakly correlated pairs to prevent representation collapse, and \(\lambda_n\) balances modality contributions. The network is trained with the Adam optimizer at a learning rate of \(5 \times 10^{-4}\) and batch size 32 using frozen DINOv3 features. To ensure fair and rigorous benchmarking, all evaluated methods undergo an identical hyperparameter search budget of 200 Bayesian optimization iterations, completing training in under two hours per model on a single NVIDIA GeForce RTX 5090 GPU.
Key Experimental Results¶
Main Results¶
The authors evaluated UniM2 on NYU-Depth-v2 (indoor RGB-D with 13 semantic classes) and MFNet (urban RGB-Thermal with 8 classes). As summarized in Table 2 from the original paper, existing USS methods suffer marked drops upon naively integrating auxiliary modalities, whereas UniM2 consistently achieves superior performance.
Original Table 2: Quantitative Comparison on NYU-Depth-v2 and MFNet Datasets
| Method | Modality | Backbone | NYU-Depth-v2 mIoU (%) | NYU-Depth-v2 Acc. (%) | MFNet mIoU (%) | MFNet Acc. (%) |
|---|---|---|---|---|---|---|
| DINOv3 K-means | RGB | ViT-S/16 | 11.1 | 26.2 | 20.1 | 67.3 |
| DINOv3 K-means | RGB + X | ViT-S/16 | 9.4 (↓1.7) | 25.5 (↓0.7) | 19.8 (↓0.3) | 66.5 (↓0.8) |
| STEGO | RGB | ViT-S/16 | 28.8 | 52.0 | 32.2 | 72.1 |
| STEGO | RGB + X | ViT-S/16 | 25.3 (↓3.5) | 45.9 (↓6.1) | 31.3 (↓0.9) | 74.9 (↑2.8) |
| EAGLE | RGB | ViT-S/16 | 27.4 | 51.6 | 34.7 | 79.6 |
| EAGLE | RGB + X | ViT-S/16 | 20.1 (↓7.3) | 40.0 (↓11.6) | 31.1 (↓3.6) | 77.4 (↓2.2) |
| UniM2 (Ours) | RGB + X | ViT-S/16 | 36.9 (↑8.1) | 56.1 (↑4.1) | 35.2 (↑3.0) | 81.5 (↑9.4) |
| DINOv3 K-means | RGB | ViT-B/16 | 14.3 | 32.8 | 20.0 | 72.1 |
| DINOv3 K-means | RGB + X | ViT-B/16 | 10.4 (↓3.9) | 23.3 (↓9.5) | 21.2 (↑1.2) | 72.4 (↑0.3) |
| STEGO | RGB | ViT-B/16 | 31.7 | 55.1 | 35.9 | 73.6 |
| STEGO | RGB + X | ViT-B/16 | 31.1 (↓0.6) | 49.7 (↓5.4) | 32.5 (↓3.4) | 74.1 (↑0.5) |
| EAGLE | RGB | ViT-B/16 | 30.9 | 49.5 | 37.8 | 72.5 |
| EAGLE | RGB + X | ViT-B/16 | 25.8 (↓5.1) | 46.9 (↓2.6) | 33.5 (↓4.3) | 74.5 (↑2.0) |
| UniM2 (Ours) | RGB + X | ViT-B/16 | 38.1 (↑6.4) | 58.8 (↑3.7) | 45.7 (↑9.8) | 76.1 (↑3.7) |
Note: Variations (↑, ↓) denote performance changes relative to the unimodal RGB STEGO baseline under the corresponding backbone.
Ablation Study¶
Ablations on NYU-Depth-v2 (ViT-S/16) rigorously analyze individual modules, anchor positioning, and fusion strategies.
Original Tables 5, 6, and 7: Component Ablation, Anchor Positioning, and Fusion Operators
| Ablation Dimension | Configuration | mIoU (%) | Pixel Acc. (%) | Description & Insights |
|---|---|---|---|---|
| Components (Tab. 5) | Baseline (Multi-modal STEGO + Conv) | 25.0 | 48.1 | Naive multimodal baseline suffering from fusion degradation |
| + CMCS + MSN (w/o CMH) | 31.3 | 50.7 | Consensus distillation outperforms unimodal baseline (28.8%) | |
| + CMCS + CMH (w/o MSN) | 34.2 | 54.3 | Decoupling contradictory gradients yields a substantial boost | |
| Full UniM2 (CMCS + MSN + CMH) | 36.9 | 56.1 | Harmonious synergy achieves the highest accuracy | |
| Anchor Positioning (Tab. 6) | No anchors (Both branches buffered) | 19.8 | 50.8 | Lacks stable reference, causing representation drift |
| Depth only (Depth as sole anchor) | 27.6 | 45.3 | Depth lacks high-level semantic richness | |
| Both (Symmetric rigid anchor) | 31.3 | 50.7 | Rigid mutual alignment fails to resolve sensor conflict | |
| RGB only (RGB anchor, Ours) | 36.9 | 56.1 | Optimal balance between semantic anchor and geometric guidance | |
| Fusion Operator (Tab. 7) | Max Operation | 30.5 | 50.2 | Hard selection discards continuous semantic cues |
| Mean Operation | 32.5 | 53.2 | Static uniform weighting cannot handle channel disparities | |
| Sum Operation | 33.0 | 53.6 | Simple addition lacks non-linear cross-channel adaptation | |
| Conv Fusion (Ours) | 36.9 | 56.1 | Adaptively extracts complementary cues when regularized by CMH |
Key Findings¶
- Reversing the Curse of Fusion Degradation: Naively adding depth or thermal inputs to STEGO or EAGLE causes mIoU to plunge (e.g., EAGLE drops from 30.9% to 25.8% on NYU-Depth-v2; STEGO drops from 35.9% to 32.5% on MFNet). In contrast, UniM2 converts cross-modal interference into solid leaps of +6.4% on NYU-Depth-v2 (38.1% vs 31.7%) and +9.8% on MFNet (45.7% vs 35.9%).
- Class-Level Geometric Disentanglement: Per-class analysis (Table 3 in the paper) reveals that geometry-sensitive categories (e.g., Sofa surges from 1.0% in RGB STEGO to 55.7% in UniM2; Table rises from 15.8% to 18.9%) leverage 3D structural cues heavily. Concurrently, appearance-dominant categories (e.g., Floor drops to 23.3% in naive depth fusion but stays preserved at 58.2% in UniM2) remain protected from gradient corruption.
- Criticality of the RGB Anchor: Abolishing the anchor entirely (No anchors) drops mIoU to 19.8%, while anchoring on depth alone yields only 27.6%. This confirms that RGB representations must serve as the primary semantic pillar in unsupervised multimodal representation learning.
Highlights & Insights¶
- First Systematic UMSS Formulation: Uncovers and formalizes the underlying mechanism behind fusion degradation in label-free settings, proving that missing label-driven arbitration leads directly to conflicting gradients.
- Asymmetric Gradient Buffering via CMH: Breaks away from conventional symmetric cross-modal distillation dogma by introducing an asymmetric, learnable buffer that effectively acts as an impedance-matching layer between disparate sensors.
- Seamless Quad-modal Scalability: Validated on the challenging MCubeS dataset (RGB, NIR, DoLP, AoLP), demonstrating steady performance scaling from single-modal (18.1%~19.1%) to multi-modal (21.8%) without structural redesign.
Limitations & Future Work¶
- Vulnerability to Low-SNR Modalities: On MCubeS, introducing the Angle of Linear Polarization (AoLP) slightly reduces mIoU from 21.8% to 20.7%, indicating that passive convolutional buffers cannot fully screen out high-noise, low-informativeness modalities without active gating.
- Reliance on Pretrained ViT Quality: The framework assumes that frozen backbones (DINOv3) produce reasonably aligned patch embeddings. For exotic sensors with strong domain shifts (e.g., FMCW radar or raw sonar), off-the-shelf ViTs may fail to provide adequate initialization.
- Future Directions: Developing SNR-aware adaptive confidence gating and extending unsupervised multi-modal distillation to 3D Gaussian Splatting and open-vocabulary scene perception.
Related Work & Insights¶
- vs STEGO [ICLR 2022]: STEGO established correspondence distillation for unimodal images. UniM2 generalizes this principle to multi-sensor setups by replacing single-image distillation with cross-modal consensus (CMCS) and resolving inter-modal conflict via CMH.
- vs EAGLE [CVPR 2024]: EAGLE utilizes eigen-aggregation for object-centric discovery. However, conflicting multi-modal spectral structures trigger catastrophic degradation in EAGLE (plunging to 20.1% on NYU-Depth-v2), whereas UniM2 maintains high manifold stability.
- vs Cross-Modal Knowledge Distillation (CMKD): Traditional CMKD enforces a strict teacher-student hierarchy with fixed directional knowledge transfer; UniM2 treats heterogeneous modalities as symmetric consensus contributors moderated through an asymmetric anchor.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering the UMSS task with an elegant, asymmetric gradient-buffering paradigm (CMCS + CMH).
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive verification across dual-modal (RGB-D, RGB-T) and quad-modal benchmarks with Bayesian hyperparameter searches.
- Writing Quality: ⭐⭐⭐⭐⭐ Exemplary clarity in dissecting the causes of fusion degradation and presenting mathematical formulations.
- Value: ⭐⭐⭐⭐⭐ Establishes a solid, highly reproducible baseline for unsupervised multi-sensor perception in robotics and autonomous systems.