Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion¶

Conference: ECCV2026
arXiv: 2606.30215
Code: None
Area: Object Detection
Keywords: RGB-T Object Detection, Cross-Modality Fusion, Sparse Computation, Efficient Detection, Lightweight Detector

TL;DR¶

A sparse fusion framework, SFEDet, is proposed. It first filters candidate foreground regions across the entire image with high recall using a lightweight single-modality detector equivalent to YOLOv8-Small, and then performs precise verification and multi-step bounding box refinement via cross-modality fusion only on sparse RoIs. This achieves detection accuracy comparable to or even better than the state-of-the-art (SOTA) at approximately 1/3 of the computational cost and 1/5 of the parameter count.

Background & Motivation¶

RGB-T object detection achieves robust performance under challenging conditions such as all-weather and low-light environments by fusing complementary information from visible (RGB) and thermal infrared (TIR) modalities. Conventional methods typically adopt dual-stream heavy backbones (e.g., CSP-Darknet-Large) and perform pixel-by-pixel cross-modality feature fusion over the entire image. Although this design achieves high accuracy, it incurs significant computational overhead—typical detectors require 300 to 1100 GFLOPs, severely limiting practical deployment on edge devices and high-resolution scenarios. However, this paper discovers an interesting phenomenon through experiments: the vast majority of image regions consist of smooth backgrounds like sky and ground. Even an extremely lightweight single-modality detector like YOLOv8-Small can achieve almost the same recall as heavy detectors at a loose threshold of IoU=0.5 (both close to 100%). This indicates that dense cross-modality fusion across the entire image involves substantial redundancy; complex fusion is only required in sparse regions that may contain foreground objects.

Nevertheless, lightweight detectors suffer from two distinct drawbacks: significantly high false-positive rates (many background regions are misdetected as foreground) and severely insufficient localization accuracy (recall drops drastically at IoU=0.75). These two bottlenecks can be precisely alleviated by fusing the rich features of both RGB and TIR modalities; cross-modality features offer stronger discriminative capability to classify true targets from background noise, while multi-modal positional information assists in correcting coarse boxes.

Based on these observations, the core mechanism of this work is to decompose the detection process into two stages: the first stage quickly scans the full image with a lightweight model to identify potential candidate regions with high recall, and the second stage performs detailed cross-modality fusion analysis and refinement only on the remaining sparse candidate regions. Core Idea: A sparse fusion mechanism called SFEDet is proposed, which first filters candidate RoIs using lightweight dual single-modality RPNs, and then performs multi-step classification and box refinement driven by Differential-Enhanced Grouped Fusion (DEGF) exclusively on these sparse RoIs, concentrating the heavy computation on the most informative foreground areas to achieve Pareto efficiency and accuracy.

Method¶

Overall Architecture¶

SFEDet is a two-stage detection framework. The first stage uses two independent YOLOv8-Small detectors to process RGB and TIR images respectively, each outputting initial detection proposals. The union of the detection results from both modalities is taken as the candidate set, and redundant proposals are removed via NMS to obtain a sparse set of non-overlapping RoIs. Subsequently, corresponding RGB and TIR feature pairs are extracted for each RoI from multi-stage FPN features using RoI Align. The core of the second stage is the Fusion-driven Evaluation and Refinement (FER) module, which only processes the sparse RoI features generated in the first stage. Through multi-step feature enhancement (grouped fusion + differential enhancement + low-level auxiliary feature injection), classification, and box regression, it progressively eliminates false positives and enhances localization accuracy.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB Image"] --> B["Lightweight RPN RGB"]
    C["TIR Image"] --> D["Lightweight RPN TIR"]
    B --> E["RoI Union + NMS"]
    D --> E
    E --> F["RoI Align<br/>Multi-level FPN Features"]
    F --> G["Differential-Enhanced Grouped Fusion<br/>DEGF"]
    G --> H["Auxiliary Feature Enhancement<br/>Low-level Backbone Features"]
    H --> I["FER Step 1<br/>Classification + Box Regression"]
    I --> J["Re-extract Auxiliary Features<br/>(Updated RoI)"]
    J --> K["FER Step 2<br/>Classification + Box Regression"]
    K --> L["Final Detection Results"]

The computational complexity of the FER module depends only on the number of RoIs rather than the input image size. Therefore, SFEDet exhibits excellent sub-linear computational scaling when processing high-resolution images. On the LLVIP dataset, when the image area increases by 4 times, FLOPs only increase by approximately 2.4 times, whereas dense fusion methods scale by over 3.8 times.

Key Designs¶

1. Sparse Fusion Mechanism: Lightweight Single-Modality RPN Filters Foreground, Fusion Module Processes Only Sparse RoIs

The core design concept stems from a simple observation: since most regions in an image are smooth backgrounds and lightweight detectors can already cover these simple regions with extremely high recall, it is unnecessary to perform complex cross-modality fusion over the entire image. In practice, the first stage of SFEDet utilizes two independent YOLOv8-Small single-modality detectors as Region Proposal Networks (RPNs) to generate initial bounding boxes from RGB and TIR images respectively. Since the information in both modalities is complementary—for instance, traffic lights are more prominent in RGB while pedestrian contours are clearer in TIR—taking the union maximizes recall. After applying NMS to the union to regulate proposal sparsity, the second stage extracts features and performs fusion inference strictly on the remaining sparse RoIs. The direct advantage of this design is that the computational complexity shifts from being linear with the image area to linear with the number of RoIs. In practical testing on high-resolution M3FD images, only about 10-30 RoIs need to be processed per frame, saving more than 60% of computation compared to full-image dense fusion.

2. Differential-Enhanced Grouped Fusion (DEGF): Lightweight and Efficient Adaptive Cross-Modality Fusion Strategy

The FER module implements a mechanism of grouped weighted fusion combined with differential enhancement to balance fusion effectiveness and computational overhead. First, the RGB and TIR RoI features are split into several groups (default is 4) along the channel dimension. Each group independently predicts fusion weights which are normalized via softmax, and weights are shared within each group to restrict computation. The benefit of grouped fusion is that the fusion weights of each group can adapt to the feature distribution of that channel subset, which is more efficient than channel-independent prediction and more granular than a single global weight. The fused features are then subjected to a difference calculation with the original single-modality features to learn an adaptive residual enhancement representation from the differential signals \(\mathbf{F}_{fu}^1 - \mathbf{F}_{fpn}^M\). The original features of each modality are multiplied by the attention weights predicted from the differential signals, added to the fusion results, and passed through LayerNorm to obtain the final enhanced features. Compared with concatenation followed by convolution, the difference operation is a lightweight computation; compared with element-wise addition, the non-linear adaptability brought by difference is superior. Ablation studies demonstrate that this design yields a ~1.2 point improvement in AP50 with almost zero additional FLOPs.

3. Rolling Convolution: Efficient Inter-Group Information Interaction via Channel Rolling

All convolutional layers in the FER module are replaced with a lightweight operator termed Rolling Convolution. Standard Group Convolution splits input channels into independent groups without information interaction between them, which degrades performance when cascaded. Rolling Convolution improves on this by running two parallel group convolution branches: one is a standard group convolution, while the other first rolls input channels along the channel dimension by half the group size before executing the group convolution. The roll operation interleaves the channels at the boundaries of different groups, allowing features that originally belonged to different groups to meet within the receptive field of the convolutional kernel. The outputs of the two branches are fused via \(1 \times 1\) pointwise convolution. Compared with standard convolution, Rolling Convolution has fewer parameters but stronger network depth and non-linear expressive capacity. In ablation studies, it achieves superior mAP compared to both depthwise separable convolution and ShuffleNet blocks, while maintaining a computational cost only slightly higher than the latter two. Compared with standard Conv2d, Rolling Convolution reduces FLOPs by 27% with nearly identical mAP (61.0 vs 61.1).

4. Step-by-Step Bounding Box Refinement and Denoising Training: Addressing Low Localization Quality of Lightweight RPNs

Bounding boxes produced by lightweight RPNs are often inaccurate in localization. Therefore, the FER module adopts a two-stage progressive refinement strategy. It first performs a round of classification and box regression based on the initial RoI features, and then re-extracts auxiliary features from the lower-layer feature maps of the backbone using the updated boxes to conduct a second round of classification and box regression. Since RoI features lose the stride scale information inherent in multi-scale feature maps, bounding box decoding utilizes the scale of the proposal boxes themselves for normalization—scaling the predicted distance residuals \(d_i\) by the width/height of the proposal boxes back to the original image space: \(D_i = \frac{x_2 - x_1}{L} \cdot d_i\) (for left and right edges) or \(D_i = \frac{y_2 - y_1}{L} \cdot d_i\) (for top and bottom edges), where \(L=16\) represents the theoretical maximum distance of the prediction head.

Regarding training, the denoising training strategy injects \(N_{de}=10\) sets of noisy variants of ground-truth boxes into the proposal list generated by the RPNs during each iteration. The noise is generated from a uniform distribution \(\delta \sim U(-0.4W, +0.4W)\). These noisy positive and negative samples force the FER module to learn to handle difficult samples with large localization errors. Ultimately, even if the IoU upper bound of the boxes output by the RPNs drops to 0.5, the FER module (without retraining) still maintains stable detection performance. Without denoising training, AP50 drops by 1.8 points.

Loss & Training¶

Training is conducted in two stages: first, the two single-modality RPNs are pre-trained independently (standard YOLOv8 configuration), followed by end-to-end joint training of the entire SFEDet. During joint training, the learning rate of the RPNs is multiplied by a decay factor of \(\lambda=0.005\) to stabilize fine-tuning. The total loss is the sum of the losses from the two RPNs and the two-step outputs of the FER, with each loss item inheriting the YOLOv8 configuration: Binary Cross Entropy (BCE) for classification, and CIoU loss + DFL loss for box regression, with respective weights \(\alpha=0.5, \beta=7.5, \gamma=0.375\). Training also employs ExpMomentum EMA, linear learning rate warmup, and data augmentations such as randomly adding Gaussian, contrast, or blur noise to single modalities.

Key Experimental Results¶

Main Results¶

The following table compares the performance of SFEDet with dense detectors configured with different fusion strategies on the M3FD dataset. SFEDet achieves 61.0 mAP / 89.8 AP50 with 24.5M parameters and 112.6G FLOPs, saving 37.6% computation with improved AP50 compared to the FPN-level dense fusion baseline (180.4G FLOPs, 88.4 AP50).

Fusion Strategy	Backbone	Params	mAP	AP50	FLOPs
Image-level Fusion	1×CSP-L	51.2M	62.5	89.8	268.0G
Backbone-level Fusion	2×CSP-S	18.8M	55.6	85.5	88.0G
FPN-level Fusion (+DEGF)	2×CSP-S	18.8M	59.6	88.4	180.4G
SFEDet (Ours)	2×CSP-S	24.5M	61.0	89.8	112.6G

On the FLIR dataset (640×512), SFEDet achieves 43.0 mAP / 81.7 AP50 with 69.0G FLOPs. On the high-resolution LLVIP dataset (1280×1024), it achieves 65.9 mAP / 96.8 AP50 with 163G FLOPs. When the image area scales by 4 times, FLOPs increase by only 2.4 times, highlighting the unique advantage of sparse fusion under high-resolution scenarios.

Ablation Study¶

Configuration	mAP	AP50	FLOPs	Description
Baseline (Element-wise Add Fusion)	57.2	85.4	104.0G	No grouped fusion/differential enhancement/auxiliary features/refinement
+ Grouped Weighted Fusion	58.4	87.1	96.4G	Grouped adaptive weights improve AP50 by +1.7
+ Differential Enhancement	58.6	87.2	96.4G	Differential adaptive weighting, no additional FLOPs
+ Auxiliary Features (Low-level Backbone)	59.9	88.5	104.6G	Low-level features are more effective than high-level FPN features
Full SFEDet	61.0	89.8	112.6G	Includes two-step refinement + rolling convolution + denoising training
- Denoising Training	59.9	88.0	111.6G	AP50 drops by 1.8, insufficient training sample diversity
- Rolling Conv (→ Standard Conv2d)	61.1	88.9	144.4G	FLOPs increase by 27%, accuracy is almost unchanged

Key Findings¶

Outstanding False-Positive Suppression: The FER module reduces the average number of false positives per M3FD image from 26.5 to 2.1, achieving a reduction of over 92%.
Excellent Localization Robustness: Even in the extreme case where the IoU bound of proposal boxes drops to 0.5 (without retraining), the AP50 of SFEDet still maintains 75.3 (M3FD), which is higher than the standard performance of many dense fusion methods.
Sub-linear Computational Scaling Brings Practical Value: On high-resolution images, the computation of dense fusion methods increases quadratically, whereas the growth for SFEDet is significantly smaller, which has direct implications for practical deployment.
Robustness to Spatial Shift: Spatial misalignment between RGB and TIR images affects both dense fusion and SFEDet, but this can be effectively mitigated through random translation data augmentation during training.

Highlights & Insights¶

"Sparse Fusion via Lightweight Proxy" Idea: The core insight is straightforward—since most of the image consists of background and lightweight models can already identify the foreground with high recall, heavy cross-modality fusion on the entire image is unnecessary. This idea has general transfer value for modern efficient multi-modal detection and can be applied to other multi-modal detection tasks such as RGB-D and RGB-NIR.
High Design Integrity: From observational validation (recall curve comparison between YOLOv8-Small and Large) to mechanism design (two-stage RPN + FER), architectural lightweight operators (Rolling Convolution), and data-level robustness enhancement (denoising training), every aspect is backed by experiments with no unverified components.
Simple and Effective Rolling Convolution: It introduces channel rolling by only one step on top of group convolution to realize inter-group interaction, which is lightweight and easy to replicate. This design can replace standard group convolution in any scenario, displaying cross-task transferability.
Convincing Analysis of False-Positive Suppression: The appendix specifically counts the number of false positives per image, showing a quantitative reduction from 26.5 to 2.1, which intuitively demonstrates the value of the FER module rather than relying solely on indirect speculation from ablation studies.

Limitations & Future Work¶

When a large number of small objects are densely distributed in a scene (e.g., crowded pedestrians), the number of RoIs increases dramatically, which weakens the savings of sparse fusion. Consequently, the computational complexity of the FER module may approach that of full-image dense fusion. The authors did not thoroughly analyze the behavior in such extreme scenarios.
The two-stage training workflow is relatively complicated: it requires first training the two RPNs independently, and then jointly fine-tuning the entire network. This imposes certain requirements on training resources and hyperparameter-tuning experience.
Currently, the RPNs employ YOLOv8-Small. Future research could explore more aggressive lightweight backbones (such as MobileNet) or incorporate distillation techniques to further compress RPN cost.
Hyperparameters such as the number of groups in rolling convolutions and RoI dimensions need tuning based on specific scenarios. Currently, a default configuration is provided, but an automatic selection mechanism guided by theory is lacking.

vs Dense Fusion RGB-T Detectors (ICAFusion, Fu-Mamba, EI2Det): Existing methods typically utilize dual-stream heavy backbones and perform dense cross-modality fusion across the entire image. Although their performance is outstanding, their FLOPs are extremely high (370-1133G). SFEDet introduces sparse fusion to RGB-T detection for the first time, achieving competitive or superior accuracy with roughly 1/3 of the computational cost.
vs Single-Modality Efficient Detectors (YOLOv8/v10, RT-DETR): Even though these methods are highly efficient, they ignore the unique computational overhead of multi-modal fusion in their designs. SFEDet explicitly demonstrates that the genuine efficiency bottleneck lies in dual-stream feature extraction and fusion, rather than the single-modality detectors themselves.
vs Deevi (Frozen Dual Backbones): Deevi reduces training cost by freezing the backbone, but still performs full-image dense fusion during inference, failing to reduce inference-time computation. SFEDet alters the computational distribution from the perspective of the inference architecture, offering deeper efficiency optimization.

Rating¶

Novelty: ⭐⭐⭐⭐ The application of sparse fusion to RGB-T detection is highly novel, and it is validated through solid observational experiments (a comparative recall analysis between lightweight and heavy models) rather than based purely on intuition.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ The method is extensively compared against competitors on three benchmark datasets (M3FD, FLIR, and LLVIP). The ablation study systematically covers each design aspect, complemented by extra experiments on convolution operator selection, hyperparameter analysis, and spatial shift robustness.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation is exceptionally clear (anchored in data analysis), the methodology is self-consistent, and the visualization is clear. Every assertion is backed by quantitative data, and the false-positive and training loss analysis in the appendix is meticulously detailed.
Value: ⭐⭐⭐⭐ This work opens a new avenue for efficient RGB-T object detection. The core concept of sparse fusion is highly valuable and easily adaptable to other multi-modal detection networks.