InfraNet: Quality-Aware RGB Guidance for Infrared Object Detection

Conference: ECCV 2026
Paper: ECCV Official
Area: Object Detection
Keywords: Infrared Object Detection, Modality-Asymmetric Learning, Quality-Aware Cross-Modal Fusion, Privileged Information Learning, Multimodal Object Detection

TL;DR

To tackle negative transfer and noise contamination from degraded RGB images in adverse conditions, InfraNet establishes an IR-centric asymmetric framework where RGB serves as quality-regulated auxiliary supervision during training via QualGate (suppressing RGB noise and compensating IR features), enabling both zero-overhead IR-only (InfraNet-IR) and high-capacity dual-modal (InfraNet-RGB-IR) inference.

Background & Motivation

Robust 24/7 perception in challenging visual environments—such as night driving, extreme glare, dense fog, and adverse weather—remains a foundational requirement for autonomous navigation, perimeter surveillance, and aerial inspection. Detectors relying solely on visible spectrum RGB cameras suffer severe degradation when ambient illumination drops or fluctuates wildly. Thermal infrared (IR) sensors overcome this fundamental bottleneck by capturing emitted thermal signatures independently of ambient light, ensuring consistent object visibility across day and night. Because RGB sensors capture rich spatial details, color, and high-frequency textures under favorable lighting while IR sensors offer thermal robustness, combining them has established multi-modal detection as a dominant paradigm.

However, almost all existing multi-modal detection frameworks rely on an implicit, symmetric assumption: RGB and IR inputs are treated as equally informative partners across both training and deployment. In harsh nighttime conditions or heavy occlusion, RGB imagery often degenerates into pure noise or uninformative dark pixels. Indiscriminately fusing such degraded RGB features into the clean IR stream contaminates the reliable thermal representations, causing severe negative transfer where multi-modal models paradoxically underperform standalone IR baselines. Furthermore, conventional feature-level fusion networks enforce dual-sensor availability at inference time, rendering systems completely dysfunctional if visible cameras fail or are obscured.

To resolve this limitation, InfraNet breaks away from equal-footing fusion and establishes an asymmetric, IR-centric paradigm: thermal IR serves as the primary backbone stream, while RGB is treated strictly as reliability-controlled auxiliary guidance during training. Core idea: construct an IR-centered asymmetric perception architecture powered by QualGate, which dynamically estimates a task-oriented scalar reliability score to suppress corrupted RGB guidance and amplify IR feature compensation, supporting both zero-overhead IR-only inference and high-capacity dual-stream deployment.

Method

Overall Architecture

InfraNet provides a unified, plug-and-play architecture for modern multi-scale detectors (such as YOLOv8/v9/v12 and ResNet). It is realized in two distinct deployment configurations: InfraNet-IR, a lightweight single-branch model optimized for edge deployment that uses RGB auxiliary guidance during training but runs purely on IR at test time; and InfraNet-RGB-IR, a dual-branch network retaining both modalities throughout inference for maximum representational capacity.

The framework organizes computation into an asymmetric dual-path design. The primary IR pathway processes the thermal image \(I_{\mathrm{IR}}\) through successive backbone and neck stages to produce multi-scale feature pyramids \(\{P_3, P_4, P_5\}\). The auxiliary RGB pathway takes \(I_{\mathrm{RGB}}\) through a lightweight stem and feature blocks to extract auxiliary representations. At three strategic fusion sites, QualGate modules dynamically evaluate RGB feature reliability, apply quality-aware suppression to the auxiliary cues, inject compensated multi-scale IR features, and output refined representations \(\{A_3, A_4, A_5\}\). For InfraNet-IR, the auxiliary branch and QualGate modules exist solely during training; at test time, they are pruned away entirely, leaving a pure IR detector with zero additional parameters or latency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Paired Input Images<br/>I_IR (Infrared) + I_RGB (Visible)"] --> B["Dual-Stream Backbone Extraction<br/>Primary IR (P3, P4, P5) and Auxiliary RGB"]
    B --> C["QualGate Quality-Aware Modulation<br/>Predict Task-Oriented Scalar Score q"]
    C --> D["Bi-Directional Modulation<br/>RGB Feature Suppression + IR Amplification"]
    D --> E["Multi-Scale Feature Fusion<br/>Output Quality-Aware Fused Maps (A3, A4, A5)"]
    E --> F{"Deployment Configuration"}
    F -->|InfraNet-IR Mode| G["Prune RGB & Fusion Modules<br/>Zero-Overhead Pure IR Inference"]
    F -->|InfraNet-RGB-IR Mode| H["Retain Dual-Stream Flow<br/>High-Capacity Joint Inference"]

Key Designs

1. Task-oriented scalar reliability control score: data-driven gate without manual quality labels

Unlike prior methods that rely on handcrafted illumination metrics or dense spatial attention maps, QualGate predicts a single scalar reliability control score \(q \in (0, 1)\) directly from the auxiliary feature map \(A_{in}\). The feature is pooled via global average pooling (GAP) and passed through a lightweight two-layer bottleneck network \(\mathrm{MLP}_{\mathrm{qual}}\):

\[q = \sigma\left(\mathrm{MLP}_{\mathrm{qual}}\left(\mathrm{GAP}(A_{in})\right)\right)\]

This score is trained end-to-end against the primary detection objective without requiring external perceptual image-quality annotations. When RGB inputs are informative, \(q\) remains high to pass informative texture cues; when RGB images are heavily corrupted by darkness, fog, or blur, detection gradient backpropagation autonomously drives \(q\) down to prevent loss inflation, acting as an automatic barrier against negative transfer.

2. Bi-directional modulation: joint RGB suppression and IR amplification

QualGate's core strength stems from simultaneously modulating both pathways in opposite directions rather than simply reweighting features: - RGB Suppression Gate: The auxiliary guidance is directly scaled by \(w_{\mathrm{rgb}}(q) = q\). When RGB inputs degrade, a small \(q\) shrinks the magnitude of \(A_{in}\), blocking uninformative artifacts from leaking into the detection representation. - IR Amplification Gate: When RGB signals become unreliable, the network must depend more heavily on the IR stream. An inverse reliability amplifier is defined as:

\[\alpha(q) = \mathrm{clip}(1.5 - q;\; 1.0, 1.5)\]

As \(q\) approaches 0 under severe corruption, \(\alpha(q)\) increases towards 1.5 to provide compensatory gain for the thermal features, ensuring that the primary IR representations remain robust and dominant.

3. Multi-scale IR feature injection and content gating: cross-level alignment

To enrich the auxiliary representation with structural thermal context, QualGate integrates multi-scale IR features \(\{C_k\}_{k=1}^K\) from the primary branch. Each IR scale is aligned through a channel adapter (CA) to produce \(F_k = \mathrm{CA}(C_k)\), and weighted by a content-adaptive gate \(g_k = \sigma(\mathrm{MLP}_{\mathrm{gate}}(\mathrm{GAP}(F_k)))\). The final fused auxiliary output \(A_{\mathrm{out}}\) is formulated as:

\[A_{\mathrm{out}} = w_{\mathrm{rgb}}(q) \cdot A_{in} + \sum_{k=1}^K \alpha(q) \cdot g_k \cdot F_k\]

This formulation guarantees that thermal features at different semantic levels actively guide cross-modal interaction, filtering atmospheric scatter and thermal sensor noise before feeding forward into the detection heads.

Loss & Training

During training, the detector calculates multi-task detection losses (task-aligned bounding box regression, distribution focal loss, and classification) on both the primary IR branch and the auxiliary fusion branch. The total optimization objective is formulated as:

\[\mathcal{L} = \mathcal{L}^{(\mathrm{main})}_{\mathrm{det}} + w_{\mathrm{aux}} \cdot \mathcal{L}^{(\mathrm{aux})}_{\mathrm{det}}\]

Setting the auxiliary weight to \(w_{\mathrm{aux}} = 0.25\) ensures that parameter updates remain anchored on the primary IR representations. This asymmetric loss balance prevents the IR backbone from overfitting to the presence of RGB features, allowing InfraNet-IR to seamlessly discard the auxiliary pathway at inference time without representation collapse.

Key Experimental Results

Main Results

InfraNet was evaluated on four standard RGB–IR benchmarks: LLVIP (pedestrians), FLIR-Aligned (automotive), DroneVehicle (UAV aerial), and M3FD (diverse conditions). On FLIR-Aligned, both the dual-stream InfraNet-RGB-IR and the deployment-friendly InfraNet-IR outperformed prior state-of-the-art methods like WaveMamba (ICCV 2025) while exhibiting dramatically lower inference latency and parameter counts.

Method	Mode	Backbone	Precision (%)	Recall (%)	F1-score (%)	mAP50 (%)	mAP (%)	Parameters (M)	Inference Time (ms)
WaveMamba [72]	RGB+IR	ResNet50	84.2	77.9	80.9	86.5	47.9	193.2	53.2
Ours (RGB-IR)	RGB+IR	ResNet50	83.5	80.4	81.9	88.6	54.3	73.8	14.4
Ours (IR)	IR	ResNet50	84.9	78.8	81.7	87.3	53.0	42.1	9.4
WaveMamba [72]	RGB+IR	YOLOv8	84.2	80.9	82.5	88.4	48.1	69.1	40.0
YOLOv8-IR [47]	IR	YOLOv8	75.1	65.3	69.9	72.9	38.3	43.7	22.0
Ours (RGB-IR)	RGB+IR	YOLOv8	85.8	79.9	82.8	89.4	55.2	54.3	13.4
Ours (IR)	IR	YOLOv8	84.6	80.9	82.7	88.5	54.0	28.2	12.5
Ours (RGB-IR)	RGB+IR	YOLOv12	84.3	81.7	83.0	89.3	55.8	56.8	13.2
Ours (IR)	IR	YOLOv12	85.0	79.5	82.2	87.7	53.2	24.7	12.0

On the LLVIP benchmark under the YOLOv8 backbone, Ours (RGB-IR) achieved 98.1% mAP50 and 70.5% mAP, while Ours (IR) achieved 97.3% mAP50 and 68.8% mAP, significantly outperforming the standalone YOLOv8l-IR baseline (95.2% mAP50 and 62.1% mAP).

Ablation Study

The contribution of each mechanism in QualGate was dissected on LLVIP using YOLOv8 in the InfraNet-IR configuration (corresponding to Table 5 in the paper):

Configuration / Setting	mAP50 (%)	mAP50-95 (%)	Note
Ours with Full QualGate	97.3	68.8	Full model: RGB suppression + IR amplification
w/o IR amp (\(\alpha(q) \equiv 1\))	97.1	67.2	Disabling IR amplification drops mAP by 1.6%
w/o RGB supp (\(w_{\mathrm{rgb}} \equiv 1\))	97.1	68.4	Disabling RGB suppression degrades performance under noise
Naive (\(q \equiv 1\))	96.9	67.7	Fixed unmodulated cross-modal fusion
Replace with CRLM [31]	96.9	66.7	Correlation-based reliability module underperforms
Replace with CAGF [65]	95.7	66.8	Standard cross-modal gated fusion struggles with asymmetry
HalluciDet pseudo-RGB [35]	95.2	64.9	Generating hallucinated RGB features introduces artifacts

Ablation on the auxiliary loss weight \(w_{\mathrm{aux}}\) (Table 7(b)) demonstrated that setting \(w_{\mathrm{aux}} = 0.25\) yielded the optimal balance (68.8% mAP). Completely removing auxiliary loss (\(w_{\mathrm{aux}}=0\)) dropped performance to 68.0% mAP, while excessive auxiliary weights (\(w_{\mathrm{aux}}=1.0\)) reduced performance to 68.1% mAP due to RGB over-reliance.

Key Findings

• Essential synergy of suppression and amplification: Removing either the IR amplification gate (\(\alpha(q) \equiv 1\), dropping to 67.2% mAP) or the RGB suppression gate (\(w_{\mathrm{rgb}} \equiv 1\), dropping to 68.4% mAP) validates that cross-modal auxiliary guidance requires both silencing corrupted noise and boosting the primary modality.
• Dynamic behavior under synthetic degradation: When artificial fog was progressively introduced (Level 0.2 \(\to\) 1.0 in Table 6), the learned score \(q\) autonomously decreased from 0.441 to 0.434. Consequently, InfraNet sustained an mAP of 67.8%, while the unmodulated naive baseline (\(q \equiv 1\)) collapsed from 66.9% down to 61.8% (-5.1% drop).
• Substantial small-object improvements in aerial imagery: On the DroneVehicle dataset, InfraNet-IR exceeded WaveMamba by +9.2% AP50 on Freight-cars (77.7% vs 68.5%) and +4.8% on Vans (69.3% vs 64.5%), showing that hierarchical multi-scale IR injection effectively overcomes altitude-induced atmospheric scattering.

Highlights & Insights

• Modality-asymmetric privileged learning paradigm: Moving away from symmetrical multi-modal fusion, InfraNet treats RGB as a training-only privileged supervisor. This bypasses the operational bottleneck of deploying dual cameras in resource-constrained hardware while fully harvesting cross-modal feature regularisation.
• Task-guided scalar gating with zero human annotation: Bypassing heavy handcrafted image-quality predictors, QualGate estimates a single global scalar \(q\) via a two-layer MLP driven purely by detection gradients, ensuring the gate reflects perceptual utility for detection rather than aesthetic quality.
• Dramatic inference efficiency gains: On FLIR-Aligned, InfraNet-IR with ResNet50 runs at 9.4ms with 42.1M parameters while delivering 53.0% mAP, decisively outperforming WaveMamba's 53.2ms latency, 193.2M parameters, and 47.9% mAP.

Limitations & Future Work

• Heuristic clipping in IR amplification: The IR compensation formula \(\alpha(q) = \mathrm{clip}(1.5 - q; 1.0, 1.5)\) relies on pre-defined constants (\(1.5\) and clipping bounds). Formulating a learnable or curvature-adaptive gain function could improve flexibility across varied sensor types.
• Sensitivity to spatial misalignment: The method assumes well-aligned RGB and IR image pairs during training. Spatial misregistration or parallax in uncalibrated camera rigs could backpropagate distorted geometric gradients into the primary IR backbone.
• Extension to tri-modal fusion: Exploring QualGate's quality-aware modulation across three or more modalities (e.g., Radar + LiDAR + Thermal IR) under extreme weather remains an exciting open research avenue.

Related Work & Insights

• vs WaveMamba [72] / FusionMamba [8]: WaveMamba utilizes state-space models and wavelets for symmetric dual-stream fusion. While effective under ideal conditions, it mandates dual-sensor inputs at test time and leaks visible noise into thermal features under nighttime conditions. InfraNet actively suppresses corrupted RGB features and supports zero-overhead IR-only deployment.
• vs HalluciDet [35] / LUPI: HalluciDet reconstructs hallucinated RGB features at inference time to aid detection, incurring heavy computational overhead and hallucination artifacts. In contrast, InfraNet uses real RGB only to regularize IR feature extraction during training, leaving inference purely thermal without any generative synthesis overhead.
• vs CRLM [31] / CAGF [65]: Prior reliability-learning modules designed for RGB-T tracking or segmentation evaluate symmetric cross-modal compatibility. When plugged into InfraNet's asymmetric setup, they underperform QualGate by ~2.0% mAP due to their inability to suppress and compensate simultaneously.

Rating

• Novelty: ⭐⭐⭐⭐ [The IR-centric asymmetric paradigm combined with QualGate bi-directional modulation offers an effective departure from symmetric fusion]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 4 benchmarks, 4 backbone families, synthetic perturbation stress tests, and detailed module ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, consistent figures, and self-contained narrative]
• Value: ⭐⭐⭐⭐⭐ [Provides substantial practical value for autonomous driving and edge surveillance systems needing high accuracy under adverse weather with minimal compute]

Related Papers

• [CVPR 2026] Target-Aware Invertible Encoder with Reconstruction Guidance for Infrared Small Target Detection
• [ECCV 2026] Cross-token Guidance Transformer for Weakly Supervised Object Localization
• [ECCV 2026] Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion
• [ECCV 2026] MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment
• [CVPR 2026] PaQ-DETR: Learning Pattern and Quality-Aware Dynamic Queries for Object Detection