Liquid Fusion of Heterogeneous Representations Towards General Salient Object Detection¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/cke520/LFNet
Area: Segmentation
Keywords: Salient Object Detection, Heterogeneous Representation Fusion, State Space Models, Liquid Neural Networks, Frequency Upsampling
TL;DR¶
Addressing the architectural spectral bias of single-paradigm backbones in general salient object detection, LFNet treats continuous global semantics from VMamba and fine-grained local textures from ConvNeXt as evolving states and exogenous stimuli within a continuous liquid dynamic system, achieving state-of-the-art performance across RGB, RGB-D, RGB-T, VSOD, and VDT tasks with superior parameter efficiency.
Background & Motivation¶
Salient Object Detection (SOD) focuses on localizing and segmenting the most visually prominent objects in complex visual environments. The research frontier has progressively advanced from single-modal RGB images to general SOD scenarios encompassing auxiliary modalities, including depth (RGB-D), thermal radiation (RGB-T), temporal sequence modeling in video SOD (VSOD), and tri-modal Visible-Depth-Thermal (VDT) settings. Such universal dense prediction tasks impose rigorous demands on visual representations: models must simultaneously capture comprehensive global semantic contexts for coarse localization and retain delicate high-frequency structural boundaries for precise pixel grouping. While vision Transformers achieve high accuracy via self-attention, their quadratic computational complexity severely hinders high-resolution dense predictions. Emerging State Space Models (SSMs) like Mamba offer linear complexity and long-range dependency modeling, yet existing works primarily attempt to engineer convoluted 2D scanning paths (e.g., cross-directional or nested S-shaped routes) to force a single backbone to capture multi-scale contexts.
From a signal processing perspective, every neural architecture possesses intrinsic spectral biases. Through dataset-level 2D Fast Fourier Transform (FFT) analysis on hierarchical feature representations across benchmarks, the authors discover that lightweight CNNs (ConvNeXt) and visual SSMs (VMamba) exhibit divergent and intersecting energy distribution curves across frequency bands. ConvNeXt naturally preserves high-frequency grid-based details and sharp boundary cues, whereas VMamba excels at low-to-medium frequency global contextual modeling. This structural divergence mathematically indicates that representations from CNNs and SSMs are inherently complementary; relying on a homogeneous backbone inevitably leaves representational blind spots regardless of scanning trajectory engineering.
Therefore, the critical challenge lies in harmonizing heterogeneous representations from both paradigms without incurring quadratic attention costs. Drawing inspiration from the continuous-time dynamics of Liquid Neural Networks (LNNs), the core idea of this work is to model continuous VMamba features and ConvNeXt features as evolving memory states and exogenous synaptic stimuli within a liquid dynamic system, dynamically regulating stimulus permeability via closed-form gating, and restoring spatial resolution through a dual-branch spectral-spatial saliency-guided upsampling operator.
Method¶
Overall Architecture¶
LFNet employs a top-down hierarchical encoder-decoder pipeline comprising a Heterogeneous Hybrid Encoder, Modality Fusion Blocks (MFB), Liquid Fusion Modules (LFM), and Saliency-Guided Upsampling (SGU) units. Multi-modal inputs are processed in parallel through pre-trained VMamba-Small and ConvNeXt-Pico streams across four resolution stages. Auxiliary modal signals (depth, thermal, video frames) are flexibly absorbed into the ConvNeXt branch via the MFB. In the decoding phase, multi-level heterogeneous features are aggregated by LFMs through dynamic state-stimulus interaction, progressively restored to full resolution via SGU operators, and optimized under multi-scale deep supervision.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Multi-modal Inputs<br/>(RGB / Depth / Thermal / Video frames)"] --> Enc["Heterogeneous Hybrid Encoder<br/>VMamba (Global) + ConvNeXt (Local)"]
Enc --> MFB["Modality Fusion Block (MFB)<br/>Adapts single / dual / tri-modal cues"]
MFB --> LFM["Liquid Fusion Module (LFM)<br/>State-stimulus dynamic gated balance"]
LFM --> SGU["Saliency-Guided Upsampling (SGU)<br/>Spectral complex filter + spatial edges"]
SGU --> Out["Multi-scale Saliency Maps<br/>(Deep supervision $O_1 \sim O_4$)"]Key Designs¶
1. Heterogeneous Hybrid Encoder and Modality Fusion: Decoupling Spectrum Coverage and Multi-Modal Extension
Homogeneous SSM models struggle with high-frequency spatial detail recovery without overly intricate scanning schemes that cause optimization bottlenecks. LFNet decouples this burden into a dual-branch backbone: VMamba-Small generates continuous sequence-based state-space features \(\mathbf{f}_i^v \in \mathbb{R}^{C_i \times H_i \times W_i}\) with linear complexity, while ConvNeXt-Pico supplies grid-based local inductive biases to cover high-frequency details. To accommodate auxiliary inputs at minimal computational cost, the ConvNeXt branch embeds a versatile Modality Fusion Block (MFB): it applies a \(1\times 1\) convolution for single-modal RGB, a single LFM unit for dual-modal tasks (RGB-D, RGB-T, VSOD), and a cascaded dual-LFM structure for tri-modal VDT tasks. This design avoids the quadratic computational complexity of multi-modal cross-attention while maintaining full-spectrum perceptual capability.
2. Liquid Fusion Module: Closed-Form State-Stimulus Dynamic Feature Aggregation
Static fusion strategies (such as element-wise addition, concatenation, or fixed convolutions) cannot adaptively arbitrate the balance between semantic context and spatial details across heterogeneous positions. Inspired by Closed-form Continuous-depth (CfC) networks, the LFM treats VMamba features as an evolving internal memory state and ConvNeXt features as an exogenous stimulus. First, the VMamba state is mapped to a smooth embedding \(\tilde{\mathbf{f}}_i^v\) via sequential \(1\times 1\) and \(3\times 3\) convolutions, and channel-wise modulation is applied to the ConvNeXt stimulus via spatial pooling: $\(\tilde{\mathbf{f}}_i^c = \text{Conv}_{1\times 1}(\mathbf{f}_i^c) \odot \sigma\left(\mathcal{M}(\text{AvgP}(\tilde{\mathbf{f}}_i^v)) + \mathcal{M}(\text{MaxP}(\tilde{\mathbf{f}}_i^v))\right)\)$ Next, concatenated features generate a spatial dynamic permeability gate \(\mathbf{G}_i = \sigma(\text{Conv}_{1\times 1}(\text{Conv}_{3\times 3}([\tilde{\mathbf{f}}_i^v, \tilde{\mathbf{f}}_i^c])))\). The fused output is formulated via the dynamic equilibrium equation: $\(\mathbf{f}_i = \text{Conv}_{3\times 3}\left((1 - \mathbf{G}_i) \odot \tilde{\mathbf{f}}_i^v + \mathbf{G}_i \odot \tilde{\mathbf{f}}_i^c\right)\)$ As \(\mathbf{G}_i \to 1\), the network prioritizes injecting local structural details from ConvNeXt; as \(\mathbf{G}_i \to 0\), it favors preserving continuous global semantics from VMamba, reconciling cross-paradigm architectural biases content-adaptively.
3. Saliency-Guided Upsampling: Dual-Branch Spectral-Spatial Anti-Aliasing Reconstruction
Conventional bilinear interpolation inevitably leads to high-frequency spectral aliasing and blurred object boundaries, whereas naive transposed convolutions introduce checkerboard artifacts and lose global shape coherence. SGU upsamples the fused feature \(\mathbf{f}_i\) by \(2\times\) into \(\mathbf{f}'_i\) and feeds it into parallel spectral and spatial branches. The spectral branch performs a 2D Fast Fourier Transform into complex frequency space, filters the spectrum using a learnable complex weight matrix \(\mathbf{w}_i\) to preserve global shape consistency, and maps back via Inverse FFT: $\(\mathbf{F}_i^{spec} = \text{IFFT}(\text{FFT}(\mathbf{f}'_i) \odot \mathbf{w}_i)\)$ Concurrently, the spatial branch applies stacked \(3\times 3\) convolutions to capture sharp high-frequency edge gradients. Concatenated dual-domain representations are blended with an input \(1\times 1\) residual connection, suppressing upsampling artifacts while recovering crisp target silhouettes.
Loss & Training¶
To ensure robust multi-scale discriminability, LFNet adopts deep supervision across all four decoding stages. The total training objective \(\mathcal{L}_{total}\) combines Binary Cross Entropy (BCE) and Intersection-over-Union (IoU) losses: $\(\mathcal{L}_{total} = \sum_{k=1}^4 \left(\mathcal{L}_{bce}(\mathbf{O}_k, \mathbf{GT}) + \mathcal{L}_{iou}(\mathbf{O}_k, \mathbf{GT})\right)\)$ where \(\mathbf{O}_k\) denotes the saliency prediction at stage \(k\) and \(\mathbf{GT}\) is the ground-truth mask. The model is optimized using AdamW with an initial learning rate of \(1\times 10^{-4}\) (\(5\%\) for pre-trained backbones) and weight decay of 0.05. A 5-epoch linear warmup is followed by cosine annealing decay, combined with Automatic Mixed Precision (AMP) and a gradient clipping threshold of 0.5 for stable convergence.
Key Experimental Results¶
Main Results¶
LFNet was benchmarked across five general SOD tasks. Representative results on single-modal RGB SOD (Table 1 excerpt) and tri-modal VDT SOD (Table 5 excerpt) are highlighted below:
| Task / Dataset | Method | Paradigm | Params (M) | \(S_m \uparrow\) (%) | \(F_m \uparrow\) (%) | \(E_m \uparrow\) (%) |
|---|---|---|---|---|---|---|
| RGB SOD (DUTS) | ICON-S | Transformer | 94.30 | 91.7 | 91.1 | 96.0 |
| RGB SOD (DUTS) | VST-S++ | Transformer | 74.90 | 90.9 | 89.7 | 94.7 |
| RGB SOD (DUTS) | Samba | Mamba | 49.59 | 93.2 | 93.0 | 96.6 |
| RGB SOD (DUTS) | LFNet (Ours) | Hybrid Mamba+CNN | 43.23 | 93.6 | 93.6 | 97.0 |
| RGB SOD (DUT-O) | Samba | Mamba | 49.59 | 88.9 | 85.9 | 92.2 |
| RGB SOD (DUT-O) | LFNet (Ours) | Hybrid Mamba+CNN | 43.23 | 90.4 | 88.4 | 93.8 |
| VDT SOD (VDT-2048) | MFFNet | CNN | 103.24 | 93.6 | 90.1 | 99.0 |
| VDT SOD (VDT-2048) | DWFPR | CNN | - | 93.8 | 90.1 | 99.0 |
| VDT SOD (VDT-2048) | Samba | Mamba | 60.28 | 93.8 | 91.0 | 99.0 |
| VDT SOD (VDT-2048) | LFNet (Ours) | Hybrid Mamba+CNN | 46.07 | 94.2 | 91.9 | 99.2 |
Ablation Study¶
Ablations on model architecture, fusion strategies, and upsampling operators across five datasets (Table 6 excerpt) validate each component:
| Setting Category | Configuration | DUTS (\(S_m\) / \(F_m\)) | DUT-O (\(S_m\) / \(F_m\)) | VDT-2048 (\(S_m\) / \(F_m\)) | Note |
|---|---|---|---|---|---|
| Full Model | LFNet (Full) | 93.6 / 93.6 | 90.4 / 88.4 | 94.2 / 91.9 | Dual-stream + LFM + SGU |
| A: Architecture | A1: VMamba Only | 92.0 / 91.7 | 89.5 / 87.1 | 91.4 / 87.4 | Lacks local high-frequency grid bias |
| A2: ConvNeXt Only | 84.8 / 82.5 | 83.3 / 78.3 | 86.8 / 80.9 | Lacks global context, severe drop | |
| A3: Dual Stream Baseline | 92.5 / 92.2 | 89.8 / 87.5 | 93.0 / 90.5 | Simple fusion without dynamic gating | |
| B: Fusion Strategy | B1: Additive Fusion | 92.7 / 92.6 | 89.9 / 87.8 | 93.5 / 90.9 | Static addition causes cross-scale aliasing |
| B2: Concatenation | 93.0 / 92.8 | 90.1 / 88.0 | 93.7 / 91.2 | Lacks state-aware filtering | |
| B3: Cross-Attention | 93.2 / 93.1 | 90.2 / 88.1 | 94.0 / 91.6 | Heavy computation, inferior to LFM | |
| C: Upsampling Strategy | C1: Bilinear Interpolation | 92.9 / 92.8 | 89.8 / 87.7 | 93.2 / 90.9 | Spectral aliasing blurs thin boundaries |
| C2: Transposed Conv | 93.2 / 93.1 | 90.1 / 88.0 | 93.5 / 91.3 | Local learning without global spectral cues |
Key Findings¶
- Crucial Complementarity of Heterogeneous Streams: Pure ConvNeXt struggles severely on dense salient localization (\(S_m\) drops to 84.8% on DUTS), yet combining it with VMamba brings substantial gains over pure VMamba (+1.6% \(S_m\) on DUTS, +2.8% \(S_m\) on VDT-2048), affirming the empirical frequency analysis.
- Liquid Dynamics Outperforms Cross-Attention: LFM surpasses heavy cross-attention (B3) across all datasets while saving memory and compute, showing that continuous ODE-inspired state-stimulus interaction effectively arbitrates representation balance.
- Spectral Upsampling Mitigates Structural Decay: Replacing SGU with standard bilinear upsampling incurs a noticeable performance drop (0.6% to 1.0% \(S_m\)), confirming that dual-domain frequency filtering prevents topological fragmentation in complex targets.
Highlights & Insights¶
- Grounding Architecture Design in Spectral Analysis: Moving beyond arbitrary heuristic combinations, the authors demonstrate via 2D FFT energy profiles why CNNs and SSMs naturally compensate for each other's spectral blind spots.
- Spatial Adaptation of Liquid Neural Dynamics: The conversion of temporal continuous-time ODE dynamics into spatial state-stimulus interactions provides an elegant, theoretically sound closed-form fusion operator.
- Exceptional Parameter-Accuracy Trade-off: Achieving SOTA performance across five distinct benchmarks with only ~43-46M parameters (under half of Transformer alternatives like ICON-S or CATNet) provides high practical deployment utility.
Limitations & Future Work¶
- Author-Admitted Limitations: The architecture currently relies on disjoint pre-trained backbones; unified pre-training for heterogeneous state-stimulus models on large-scale segmentation datasets remains unexplored.
- Identified Limitations: Deploying dual-branch networks on specialized edge hardware requires supporting two divergent operator libraries (selective scan kernels and depthwise convolutions), which may impact runtime latency optimization on certain embedded accelerators.
- Future Directions: The liquid fusion paradigm can be generalized to broader dense prediction tasks such as camouflaged object detection, medical image segmentation, and image restoration tasks.
Related Work & Insights¶
- vs Samba: Samba represents the pioneering general SOD framework using pure Mamba, which forces a single SSM to handle both semantics and boundaries via scanning paths; LFNet introduces heterogeneous CNN cooperation and liquid gating, achieving lower parameter counts (43.23M vs 49.59M on RGB) and higher accuracy.
- vs VSCode / VST-S++: These Transformer architectures rely on quadratic self-attention mechanisms; LFNet maintains linear computational scaling while delivering sharper object silhouettes through spectral-spatial upsampling.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ (Creative spatial adaptation of liquid neural network dynamics combined with 2D FFT spectral analysis)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Exhaustive validation across 5 distinct tasks and 15+ benchmark datasets with meticulous ablations)
- Writing Quality: ⭐⭐⭐⭐⭐ (Clear mathematical formulation, rigorous structure, and self-contained explanations)
- Value: ⭐⭐⭐⭐☆ (Offers an efficient, scalable template for heterogeneous multi-modal dense prediction architectures)