RA-SOD: Reliability-Aware RGB-T Salient Object Detection under Modality Degradation

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/zaoxienian/RA-SOD
Area: Segmentation
Keywords: Salient Object Detection, RGB-T Multimodal, Modality Degradation, Mixture of Experts, Uncertainty Estimation

TL;DR

Addressing frequent noise, blur, and sensor distortion in visible and thermal modalities, RA-SOD models modality reliability across encoding, decoding, and fusion stages using residual MoE feature modulation, uncertainty-guided dual-stream bidirectional correction, and pixel-wise modality competition, achieving state-of-the-art robustness in degraded environments.

Background & Motivation

RGB-Thermal (RGB-T) salient object detection improves upon unimodal visual saliency by combining complementary spectral information from visible and infrared sensors, delivering superior robustness under varying lighting and complex scene backgrounds. However, most existing RGB-T SOD models operate under an implicit, idealized premise: that unimodal features extracted by the encoder are intrinsically reliable, and cross-modal discrepancies only emerge at later fusion stages. Consequently, existing research has predominantly concentrated on designing intricate multi-scale interaction layers, cross-attention mechanisms, and collaborative graph networks.

In real-world deployment scenarios, sensory evidence provided by each individual modality is inherently fragile and volatile. Visible RGB imagery is frequently disrupted by adverse low-light conditions, motion blur, and severe sensor noise. Concurrently, thermal infrared imagery often encounters thermal crossover (where target and background temperatures equilibrate), drastic contrast compression, and sensor striping or hot-block artifacts. When such modality degradation occurs, standard backbones apply fixed, input-agnostic transformations that cannot adapt to input-dependent distribution shifts. As a result, corrupted representations are propagated through the network hierarchy and magnified, forcing subsequent fusion modules to operate on distorted features where valid object structures cannot be distinguished from modality-specific artifacts.

To overcome these challenges, reliability awareness must not be relegated to an afterthought or an auxiliary weighting cue, but modeled as an active principle throughout the entire representation and decoding lifecycle. Core idea: establish a holistic reliability-aware framework spanning representation, refinement, and fusion via reliability-conditioned MoE backbone modulation, uncertainty-guided dual-stream recursive refinement, and spatially adaptive pixel-wise modality competition.

Method

Overall Architecture

RA-SOD consists of three closely interconnected stages: the Reliability-Conditioned Backbone Representation (RCBR) for adaptive feature encoding, the Uncertainty-Guided Dual-Stream Decoding (UGDD) for recursive cross-modal correction, and the Pixel-wise Fusion Stream Decoding for fine-grained modality competition. Given paired RGB and thermal inputs \(I^r\) and \(I^t\), stage-wise base features are first extracted by a frozen shared backbone and modulated by a parallel residual branch with gated expert routing. In the subsequent top-down decoding hierarchy, RGB and thermal streams recursively refine features guided by explicitly predicted spatial uncertainty maps. Finally, the fusion stream arbitrates cross-modal features through pixel-wise competition, and saliency predictions from all three streams are linearly aggregated to yield the final saliency map.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Paired Input Images (RGB / Thermal)"] --> Enc["Shared Backbone Feature Extraction"]
    Enc --> RCBR["Reliability-Conditioned Backbone Representation<br/>Frozen Backbone + Gated MoE Residual Branch"]
    RCBR --> UGDD["Uncertainty-Guided Dual-Stream Decoding<br/>Spatial Uncertainty Map + Bidirectional Correction"]
    RCBR --> MFM["Multimodal Feature Fusion (MFM)"]
    MFM --> PMC["Pixel-wise Modality Competition Fusion<br/>Spatial Dynamic Softmax & Weighted Feature Routing"]
    UGDD --> PMC
    UGDD --> PredHead["Tri-Stream Saliency Heads (RGB / Thermal / Fusion)"]
    PMC --> PredHead
    PredHead --> Agg["Linear Summation to Final Saliency Map"]

Key Designs

1. Reliability-Conditioned Backbone Representation: Residual MoE modulation with frozen structural priors

Under modality degradation, severe feature distribution shifts degrade standard representations. Completely fine-tuning the backbone risks overfitting on corrupted samples or discarding universal vision priors, whereas keeping the backbone purely static limits adaptability to degraded inputs. To reconcile stability and adaptability, RCBR establishes a parallel residual modulation branch alongside a frozen backbone. At stage \(l\), the frozen backbone \(B_l(\cdot)\) produces a base feature \(\mathbf{F}_l^{m,\text{base}} = B_l(\mathbf{F}_{l-1}^m)\) for modality \(m \in \{r, t\}\). A lightweight gating network \(g_l^m\) inspects this base feature to predict expert routing probabilities: $$ \boldsymbol{\alpha}l^m = \text{Softmax}\left(g_l^m(\mathbf{F}_l^{m,\text{base}})\right), \quad \sum^m = 1 $$}^K \alpha_{l,k

In the parallel residual stream, a shared bank of expert transformations \(E_{l,k}(\cdot)\) is combined with a modality-specific low-rank adapter \(A_l^m(\cdot)\): $$ \Delta \mathbf{F}l^m = \sum}^K \alpha_{l,k}^m E_{l,k}(\mathbf{F{l-1}^m) + A_l^m(\mathbf{F}^m) $$ The final representation is formed as \(\mathbf{F}_l^m = \mathbf{F}_l^{m,\text{base}} + \Delta \mathbf{F}_l^m\). When low-light noise or contrast suppression corrupts an input modality, the base feature reflects this distributional shift, prompting the gating network to route features through specialized expert subspaces to adaptively compensate corrupted components while leaving pristine structural priors intact.

2. Uncertainty-Guided Dual-Stream Decoding: Explicit confidence gating for bidirectional error suppression

During hierarchical decoding, localized degradation (such as blown-out highlights or thermal edge blurring) can inject misleading activations. Unchecked cross-modal attention or naive concatenation inadvertently leaks noise into the uncorrupted modality stream. UGDD equips the top-down decoding hierarchy with explicit spatial uncertainty gating. At stage \(l\), preliminary decoding features \(\hat{\mathbf{D}}_l^m\) are processed by a lightweight modality-shared uncertainty predictor \(\phi_l\): $$ \mathbf{U}_l^m = \sigma\left(\phi_l(\hat{\mathbf{D}}_l^m)\right) \in [0, 1]^{H_l \times W_l} $$

The predicted map \(\mathbf{U}_l^m\) acts as a spatial unreliability signal. Cross-modal correction is strictly gated so that regions with high uncertainty absorb more restorative guidance from the peer modality: $$ \mathbf{D}_l^r = \hat{\mathbf{D}}_l^r + \mathbf{U}_l^r \odot \mathcal{T}_l(\hat{\mathbf{D}}_l^t), \quad \mathbf{D}_l^t = \hat{\mathbf{D}}_l^t + \mathbf{U}_l^t \odot \mathcal{T}_l(\hat{\mathbf{D}}_l^r) $$ where \(\mathcal{T}_l(\cdot)\) is a channel-alignment transformation. If the visible modality suffers from darkness, elevated \(\mathbf{U}_l^r\) values allow thermal cues to actively reconstruct foreground targets; conversely, when the thermal modality is ambiguous, clear visible boundary details are injected, thereby halting the propagation of erroneous signals down the decoding pyramid.

3. Pixel-wise Modality Competition: High-resolution spatial arbitration for fine-grained fusion

Even after dual-stream correction, the informational reliability of RGB and thermal modalities exhibits sharp spatial heterogeneity across different image regions. Applying scalar weighting or uniform blending often softens crisp object contours or retains background clutter. PMC introduces a spatial competition mechanism in the central fusion decoding stream. At decoding stage \(l\), the upsampled prior fusion state \(\text{Up}(\mathbf{D}_{l+1}^{fus})\) and the refined features \(\mathbf{D}_l^r, \mathbf{D}_l^t\) are evaluated by a convolutional module \(\psi_l\) to produce spatial competition weights: $$ [\mathbf{W}l^r, \mathbf{W}_l^t] = \text{Softmax}\left(\psi_l([\text{Up}(\mathbf{D}}^{fus}); \mathbf{Dl^r; \mathbf{D}_l^t])\right) $$ where \(\mathbf{W}_l^r + \mathbf{W}_l^t = 1\) is enforced at every pixel coordinate. The dynamic guidance feature is aggregated as: $$ \mathbf{H}_l = \mathbf{W}_l^r \odot \mathbf{D}_l^r + \mathbf{W}_l^t \odot \mathbf{D}_l^t + \text{Up}(\mathbf{D}) $$ By enabling per-pixel competition, the network dynamically assigns dominance to visible textures along sharp silhouette boundaries while prioritizing thermal infrared radiation across low-contrast or shadowed object interiors.}^{fus

Loss & Training

To prevent optimization from collapsing into a single dominant modality, RA-SOD imposes deep multi-stream supervision. Lightweight prediction heads \(\mathcal{O}^r, \mathcal{O}^t, \mathcal{O}^{fus}\) with identical architectures but decoupled parameters are attached to the RGB, Thermal, and Fusion decoding outputs to generate saliency maps \(\mathbf{S}^r, \mathbf{S}^t, \mathbf{S}^{fus}\).

The network is trained end-to-end using a joint binary cross-entropy (BCE) and intersection-over-union (IoU) objective across all three streams: $$ \mathcal{L} = \sum_{m \in {r, t, fus}} \left( \mathcal{L}{\mathrm{BCE}}(\mathbf{S}^m, \mathbf{G}) + \mathcal{L}) \right) $$ where }}(\mathbf{S}^m, \mathbf{G\(\mathbf{G}\) denotes the ground-truth mask. During inference, the final saliency prediction is computed via linear summation: \(\mathbf{S} = \mathbf{S}^r + \mathbf{S}^t + \mathbf{S}^{fus}\), harvesting both modality-specific and cross-modally integrated evidence.

Key Experimental Results

Main Results

RA-SOD was evaluated against modern state-of-the-art CNN-based RGB-D and RGB-T SOD architectures across standard benchmarks (VT821, VT1000, VT5000) and the severe degradation challenge benchmark VT-IMAG. The table below reports the quantitative comparisons (corresponding to Table 1 and Table 3 in the paper).

Dataset	Metric	Ours (RA-SOD)	Second Best (ConTriNet / CAVER)	Gain / Reduction
VT5000	Structure-measure (\(S_m\)) ↑	0.897	0.894 (ConTriNet)	+0.003
VT5000	Mean F-measure (\(F_\beta\)) ↑	0.870	0.860 (ConTriNet)	+0.010
VT5000	Weighted F-measure (\(F_\beta^w\)) ↑	0.855	0.846 (ConTriNet)	+0.009
VT5000	Mean E-measure (\(E_m\)) ↑	0.937	0.935 (SMR-Net) / 0.934	+0.002
VT5000	Mean Absolute Error (\(M\)) ↓	0.029	0.030 (ConTriNet)	-0.001
VT1000	Mean F-measure (\(F_\beta\)) ↑	0.915	0.905 (LAFB) / 0.903	+0.010
VT1000	Weighted F-measure (\(F_\beta^w\)) ↑	0.913	0.909 (CAVER)	+0.004
VT-IMAG	Structure-measure (\(S_m\)) ↑	0.829	0.828 (ConTriNet)	+0.001
VT-IMAG	Mean F-measure (\(F_\beta\)) ↑	0.765	0.745 (ConTriNet)	+0.020
VT-IMAG	Weighted F-measure (\(F_\beta^w\)) ↑	0.735	0.727 (ConTriNet)	+0.008
VT-IMAG	Mean E-measure (\(E_m\)) ↑	0.908	0.902 (ConTriNet)	+0.006

Ablation Study

The table below details the ablation experiments on the components of the Reliability-Conditioned Backbone Representation (RCBR) on standard benchmarks (corresponding to Table 6 in the paper).

Variant	Adapter \(A\)	Expert Bank \(E\)	VT821 (\(F_\beta^w\) / \(M\))	VT1000 (\(F_\beta^w\) / \(M\))	VT5000 (\(F_\beta^w\) / \(M\))	Note
\(B_0\) (Frozen Backbone)	×	×	0.840 / 0.032	0.906 / 0.019	0.851 / 0.030	baseline without residual modulation
\(B_1\) (+A only)	✓	×	0.843 / 0.032	0.914 / 0.018	0.853 / 0.031	modality-specific adapter only
\(B_2\) (+E only)	×	✓	0.842 / 0.032	0.910 / 0.020	0.851 / 0.029	shared MoE bank only
\(B_3\) (RA-SOD Full)	✓	✓	0.846 / 0.031	0.913 / 0.018	0.855 / 0.029	joint adapter and expert modulation

Key ablation insights on other modules include: - In Table 7, removing the UGDD module causes \(F_\beta^w\) on VT-IMAG to drop from 0.735 to 0.728, and on synthetic degradation tests (Deg.) from 0.907 to 0.897. Dual-stream decoding without explicit uncertainty estimation (No-Unc.) achieves only 0.903 on Deg., validating the role of uncertainty gating. - Table 8 confirms that the uncertainty map \(\mathbf{U}_l^m\) responds selectively to corrupted modalities: under low-light RGB degradation, \(\mathbf{U}_l^r\) escalates from 0.32 to 0.57 (while \(\mathbf{U}_l^t\) remains stable at 0.31), proving that the gating signal faithfully reflects perceptual reliability.

Key Findings

• Pronounced margins under severe degradation: While RA-SOD achieves competitive gains on clean benchmarks, its margin expands significantly on VT-IMAG—improving \(F_\beta\) by 0.020 (0.765 vs 0.745) over ConTriNet, highlighting the necessity of reliability modeling when sensors degrade.
• Optimal expert capacity at \(K=3\): Ablation across expert counts (Table 9) shows performance peaking at \(K=3\) on VT5000 (\(F_\beta^w = 0.855\)). Increasing to \(K=5\) slightly reduces \(F_\beta^w\) to 0.850 due to gate redundancy and optimization difficulty.
• Favorable parameter-speed trade-off: With a Res2Net-50 backbone, RA-SOD uses 46.72M parameters and 60.13 GFLOPs at 11.24 FPS (Table 4), far more practical than LAFB (453M parameters, 4.62 FPS).

Highlights & Insights

• Decoupled residual MoE adaptation: By separating universal semantic extraction (frozen backbone) from degradation compensation (residual MoE branch), the model prevents catastrophic forgetting while dynamically correcting sensor noise.
• Bidirectional uncertainty-gated filtering: Rather than unconstrained cross-attention, uncertainty maps act as localized filters, ensuring information transfers only into ambiguous regions from clear peer modalities.
• Broad cross-task transferability: The three-stage paradigm—adaptive encoding, uncertainty-guided decoding, and pixel-wise competition—can be directly applied to other degraded multimodal tasks, including RGB-D depth sensor failures, event cameras, and multispectral autonomous navigation.

Limitations & Future Work

• Frozen backbone capacity bounds: Freezing large portions of the backbone prioritizes generic visual representations, but may constrain expressiveness when encountering out-of-distribution thermal radiation distributions.
• Sensitivity to spatial alignment: Pixel-wise competition implicitly assumes pixel-accurate geometric alignment between RGB and thermal sensors. Mechanical vibration or parallax in non-coaxial camera setups could introduce spatial misalignment.
• Future research: Extending reliability modeling to uncalibrated or weakly aligned multispectral setups, and integrating dynamic uncertainty assessment into multimodal large vision models (VLM).

Related Work & Insights

• vs ConTriNet [35]: ConTriNet established the triple-flow paradigm but assumed input features were uniformly trustworthy. RA-SOD builds upon multi-stream decoding by introducing RCBR residual gating and UGDD uncertainty correction, outperforming ConTriNet on VT-IMAG by 2.0% in \(F_\beta\).
• vs CAVER [29] & SMR-Net [59]: CAVER uses cross-modal view-mixed Transformers and SMR-Net relies on semantic reinforcement, but both are prone to mistaking thermal hot-spots or low-light noise for salient targets. RA-SOD's explicit uncertainty suppression and pixel competition effectively eliminate these false positives.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Systematically tackles modality degradation throughout encoding, decoding, and fusion with elegant MoE and uncertainty gating designs)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Exhaustive benchmarking over 4 datasets, synthetic degradation suites, efficiency comparisons, and extensive ablations)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clear problem framing, well-formulated methodology, and cohesive narrative)
• Value: ⭐⭐⭐⭐☆ (Highly relevant for reliable multi-sensor perception under real-world adverse atmospheric and lighting conditions)

Related Papers

• [CVPR 2026] Uncertainty-Aware Modality Fusion for Unaligned RGB-T Salient Object Detection
• [ECCV 2026] ModuSeg: Decoupling Object Discovery and Semantic Retrieval for Training-Free Weakly Supervised Segmentation
• [ICLR 2026] S3OD: Towards Generalizable Salient Object Detection with Synthetic Data
• [ECCV 2026] PanoSAM2: Lightweight Distortion- and Memory-aware Adaptions of SAM2 for 360 Video Object Segmentation
• [ECCV 2026] ET-SAM: Efficient Point Prompt Prediction in SAM for Unified Scene Text Detection and Layout Analysis