Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction¶

Conference: ECCV 2026
arXiv: 2606.26812
Code: https://github.com/ixilai/AMG-Fuse
Area: Multi-modal VLM
Keywords: Multi-modal image fusion, Infrared and visible image fusion, Adverse weather image restoration, Mask-guided learning, Cross-modal attention

TL;DR¶

AMG-Fuse proposes a mask-guided multi-modal image fusion framework for adverse weather. By decoupling modal contribution masks from the "Pseudo Ground Truth" (Pseudo GT) and combining the Mask-guided Feature Extraction Module (MFEM) with Mask-guided Cross-modal Cross-Attention (MCCA), Mask-Guided Learning Strategy (MGLS), and Task-coupled Degradation-Aware learning Strategy (TDAS), it simultaneously achieves feature restoration and cross-modal interaction in a unified network. It comprehensively outperforms SOTA methods under rain, snow, and haze, as well as in real-world scenarios, while achieving optimal mAP on downstream object detection tasks.

Background & Motivation¶

Multi-Modal Image Fusion (MMIF) integrates complementary information from different modalities to generate richer scene representations. Taking Infrared and Visible Image Fusion (IVIF) as an example, visible images capture textural details, while infrared images highlight salient targets via thermal radiation. Their fusion serves downstream tasks such as object detection and semantic segmentation. Existing IVIF methods primarily operate under ideal circumstances, but real-world adverse weather (snow, rain, haze) causes severe image degradation, disrupting feature representations and making it difficult for the fusion network to simultaneously achieve the dual goals of 'feature restoration' and 'cross-modal complementation'.

Existing solutions addressing adverse weather suffer from inherent limitations. The first category is the two-stage "restoration-then-fusion" paradigm, which processes each modality using independent restoration networks before feeding them into the fusion network. This introduces two issues: the restoration phase focuses on intra-modal recovery while the fusion phase focuses on inter-modal complementation, and the inconsistent optimization objectives lead to unstable training; additionally, artifacts generated in the restoration stage propagate to the fusion stage, causing cumulative errors. The second category uses "Pseudo Ground Truth" (Pseudo GT) as supervision—where the fusion results generated by existing fusion methods on clean image pairs serve as training targets. This simplifies the optimization objective to \(\theta^* = \arg\min_\theta \|f(VI, IR; \theta) - GT_{Pse}\|_1\), reducing optimization complexity and preserving global structure and details. However, the pseudo ground truth itself carries information loss and modal bias, causing the network to easily overfit to the static pixel distribution of the pseudo GT rather than learning dynamic modal allocation mechanisms, which may even discard crucial thermal target information from infrared images in clean scenes.

Key Challenge: While pseudo ground truth simplifies training, it inclines the network to mechanically replicate superficial features, ignoring the dynamic excavation of cross-modal complementary cues. Core Idea: Deriving a modal contribution mask \(M\) (satisfying \(Fuse = M \times VI + (1-M) \times IR + \varepsilon\)) from the mapping relationship between the pseudo ground truth and source images. Treating this mask as an external prior to explicitly model cross-modal interactions forces the network to learn the allocation mechanism of "how much each modality contributes to the pseudo ground truth" rather than simply copying the pixel values of the pseudo ground truth.

Method¶

Overall Architecture¶

The input to AMG-Fuse is a pair of degraded infrared image \(IR\) and visible image \(VI\), and the output is a restored and fully fused image. The entire network features an end-to-end single-stage structure. The core idea is to embed the modal contribution mask representation into the feature extraction and interaction processes, allowing the network to dynamically allocate fusion weights of different modalities while restoring degraded features.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["IR + VI<br/>(Degraded Input)"] --> B["Conv + ResBlock<br/>Shallow Feature Extraction"]
    B --> C["ResBlock Refinement<br/>Modality-specific Features"]
    B --> D["HTB Histogram Transformer<br/>Intensity-partitioned Self-Attention"]
    C --> E["Mask-guided Feature Extraction MFEM<br/>Mask Construction → MCCA Cross-Attention"]
    D --> E
    E --> F["Fused Image Output"]

Specific workflow: Conv and residual blocks are first employed to extract multi-modal shallow features, which are then combined to generate initial fused features. Modality-specific features are further refined via residual blocks, whereas the fused features are fed into a Histogram Transformer Block (HTB). HTB partitions spatial features based on pixel intensity and performs self-attention within each partition, which effectively captures similar degradation patterns (e.g., rain streaks) across long ranges. Finally, the refined multi-modal features and the HTB-enhanced fused features together enter the Mask-guided Feature Extraction Module (MFEM) to perform mask construction and Mask-guided Cross-modal Cross-Attention (MCCA), producing the final fused image.

During training, AMG-Fuse utilizes pseudo ground truth \(GT_{Pse}\) (generated by EMMA on clean image pairs) as auxiliary supervision, optimized jointly with MGLS, TDAS, gradient loss, and color consistency loss. Among these, the weight coefficient \(\lambda\) of MGLS gradually decays as training epochs progress, allowing the pseudo ground truth to stabilize training in the early stages and yield to the supervision of clean source images in the later stages.

Key Designs¶

1. Modal Contribution Mask Construction: Decoupling Modal Allocation Weights from Fusion Formulation

The core goal of IVIF is to suppress cross-modal redundancy and extract complementary information. Ignoring feature enhancement and dimensionality reduction, the fusion result can be expressed as \(Fuse = M \times VI + (1-M) \times IR + \varepsilon\), where \(M\) represents the modality weight allocation mask, and \(\varepsilon\) denotes error and noise terms. When \(Fuse\), \(VI\), and \(IR\) are known, directly solving \(M = \frac{Fuse - IR}{VI - IR} + \varepsilon_M\) under adverse weather encounters severe issues: overexposed visible regions in hazy/snowy weather and low contrast in infrared images cause the denominator \(VI - IR\) to be dominated by degradation factors rather than semantic content, rendering the mask incapable of reflecting the true modal distribution; in night scenes, \(VI - IR\) is close to zero or negative over large areas, causing numerical instability.

This work introduces a critical modification by incorporating the \(Fuse\) information into the denominator, rewriting it as \(M = \frac{Fuse - IR}{VI - IR + Fuse} + \varepsilon_M\). This operation achieves two goals with one stroke: since \(Fuse\) is optimized and degraded features are restored, its structure is more stable, thereby suppressing the erroneous amplification of masks caused by visible brightness bias; simultaneously, it avoids numerical instability from denominators approaching zero. Performing feature decomposition on pseudo ground truths using this mask yields the visible contribution map \(F_{VI} = M_{Pse} \times VI_C\) and the infrared contribution map \(F_{IR} = (1-M_{Pse}) \times IR_C\), which visually demonstrates the allocation strategies of different fusion algorithms for each modal information.

2. Mask-guided Feature Extraction Module (MFEM) and Mask-guided Cross-modal Cross-Attention (MCCA)

Direct training on pseudo ground truths forces the network to duplicate the inherent bias of the pseudo GT, emphasizing the reproduction of superficial features while disregarding effective cross-modal interaction. The design of MFEM follows the synthesis paradigm of the formulation \(Fuse = M \times VI + (1-M) \times IR\): multi-modal features are first fed into Residual Squeeze Blocks (RSB) for refinement, and fused features are enhanced by HTB; then, the mask \(M\) is constructed based on both; finally, the network learns the multi-modal interaction patterns residing in the pseudo ground truth through the MCCA module.

MCCA adopts a cross-attention mechanism: multi-modal features serve as Query guided by mask-based weighting, while the fused features act as Key and Value. Specifically, the mask weights the visible query \(Q_{VI}\) and infrared query \(Q_{IR}\) respectively, guiding the network to focus on prominent features in each modality. Depth-wise separable convolutions are introduced into both Query and Key-Value branches to expand local receptive fields. Performing cross-attention between the mask-weighted query and the key/value of fused features decouples and recombines multi-modal features in the fusion space—essentially enabling the network to learn "how much visible and infrared information should be extracted at this position during fusion" rather than blindly copying the pseudo ground truth.

3. Mask-Guided Learning Strategy (MGLS): Supervising Dynamic Fusion with Modal Distribution of Pseudo Ground Truth

The core idea of MGLS is to constrain the network using the modal allocation patterns from the pseudo ground truth. First, the mask \(M_{Pse}\) is calculated based on formula (5) using the pseudo ground truth \(GT_{Pse}\) and clean source images \(VI_C, IR_C\). Visible allocation map \(F_{VI}\) and infrared allocation map \(F_{IR}\) are decoupled as \(F_{VI} = M_{Pse} \times VI_C\) and \(F_{IR} = (1-M_{Pse}) \times IR_C\). Then, an L1 loss is applied to the corresponding modal features \(\hat{F}_{VI}\) and \(\hat{F}_{IR}\) output by the network:

\[\mathcal{L}_{MGLS} = \frac{1}{HW} (\|\hat{F}_{VI} - F_{VI}\|_1 + \|\hat{F}_{IR} - F_{IR}\|_1)\]

This forces the network to learn the distribution of "how much each modality contributes to the pseudo ground truth", rather than simply fitting the global appearance of the pseudo ground truth. Crucially, the weight coefficient \(\lambda\) of MGLS decays during training: \(\lambda\) is larger in the early stages to utilize pseudo ground truth for stable training and fast convergence, and decreases progressively in later stages to let the supervision of clean source images dominate the optimization, preventing the network from being locked into the bias of pseudo ground truth.

4. Task-coupled Degradation-Aware learning Strategy (TDAS): Using Restoration Model as a "Cleanliness Detector"

The design motivation of TDAS stems from a key observation: the mask \(M_{Deg}\) under degraded scenarios can effectively capture the distribution of degraded regions—for instance, \(M_{Deg}\) successfully suppresses most rain streaks in rainy scenes. In \(Fuse = M_{Deg} \times VI_{Deg} + (1-M_{Deg}) \times IR_{Deg} + \varepsilon\), since \(Fuse\) is a degradation-free image, \(VI_F = M_{Deg} \times VI_{Deg}\) should represent the degradation-free visible component.

Based on this, TDAS introduces a pre-trained restoration model \(\mathcal{R}(\cdot)\) as a "cleanliness detector": if \(VI_F\) has been fully restored, the output of the restoration model \(\mathcal{R}(VI_F)\) should approximate an identity mapping (since the input itself is already clean). The TDAS loss is defined as:

\[\mathcal{L}_{TDAS} = \frac{1}{H \times W} \|VI_F - \mathcal{R}(VI_F)\|_1\]

When the fusion network successfully restores clean features, \(VI_F\) is close to \(\mathcal{R}(VI_F)\), yielding a small loss; otherwise, if the fusion network fails to remove degradations (e.g., remaining rain streaks), the restoration model will perform deraining on it, causing a change in output and an increased loss. This acts as an auxiliary supervision signal using the restoration task to enhance the network's degradation awareness, guiding it to prioritize restoring clear and salient areas. TDAS is complementary to MGLS: TDAS focuses on "whether the features are clean", whereas MGLS focuses on "how modalities are allocated".

Loss & Training¶

Beyond MGLS and TDAS, AMG-Fuse introduces two source image supervision losses to enhance the model's capability in capturing multi-modal distributions. Gradient loss preserves structural details: \(\mathcal{L}_{grad} = \frac{1}{HW} \|\nabla Fuse - \max(|\nabla VI_C|, |\nabla IR_C|)\|_1\), demanding that the gradient intensity of the fused image is no less than the maximum of the two source images. Color consistency loss maintains color distribution: \(\mathcal{L}_{color} = \frac{1}{HW} \|F_{CbCr}(Fuse) - F_{CbCr}(VI_C)\|_1\), constraining the color components of the fused result to match those of the visible image in the CbCr color space.

The total loss is \(\mathcal{L}_{total} = \lambda \times \mathcal{L}_{MGLS} + \mathcal{L}_{TDAS} + \mathcal{L}_{color} + \mathcal{L}_{grad}\), where \(\lambda\) is a decaying coefficient that decreases progressively with training epochs; the weights of the remaining three terms are set to 1, requiring no fine-tuning. Training utilizes 1,000 images each for snow, rain, and haze from the AWMM-100k dataset as the training set, cropped into \(168 \times 168\) patches. The Adam optimizer is used with an initial learning rate of \(1\times 10^{-3}\) and a batch size of 2, trained on 4 RTX 3090 GPUs for 200 epochs.

Key Experimental Results¶

Main Results¶

Under adverse conditions of snow, rain, and haze, the model is compared against 7 baseline methods (LRRNet, Text-DiFuse, EMMA, Text-IF, GIFNet, SAGE, AWFusion). Except for AWFusion, which handles degradation end-to-end, all other methods employ AdaIR as a pre-restoration module. AMG-Fuse ranks in the top two across the majority of metrics.

Scenario	Method	\(Q_M\uparrow\)	\(Q_G\uparrow\)	SSIM\(\uparrow\)	\(VIF\uparrow\)	\(Q^{AB/F}\uparrow\)
Snow	EMMA (CVPR 24)	0.5353	0.4235	0.4096	0.3252	0.5470
Snow	Text-IF (CVPR 24)	0.5942	0.4199	0.4298	0.3300	0.5419
Snow	AWFusion (INFFus 26)	0.5535	0.3552	0.3734	0.3024	0.4937
Snow	AMG-Fuse	0.6408	0.4240	0.4302	0.3463	0.5519
Rain	EMMA (CVPR 24)	0.5871	0.3745	0.3700	0.3384	0.4866
Rain	Text-IF (CVPR 24)	0.6375	0.4085	0.4046	0.3489	0.5149
Rain	AMG-Fuse	0.6940	0.4163	0.4080	0.3582	0.5184
Haze	EMMA (CVPR 24)	0.4149	0.3553	0.3296	0.2559	0.4617
Haze	AWFusion (INFFus 26)	0.5471	0.4283	0.3901	0.3433	0.5322
Haze	AMG-Fuse	0.5438	0.4376	0.4083	0.3445	0.5414

In real-world scenes (real-world haze data), AMG-Fuse ranks in the top two across all 7 metrics (\(Q_{MI}\): 0.7234, \(Q_G\): 0.5114, \(Q_M\): 1.1022, \(VIF\): 0.4373, SSIM: 0.5202, \(Q^{AB/F}\): 0.5957).

Downstream Tasks: For object detection using YOLOv7 on the M3FD dataset, AMG-Fuse achieves the optimal [email protected] of 0.843 and [email protected]:0.95 of 0.541, ranking in the top two for 5 out of 6 categories.

Outperforming Pseudo GT Generator: AMG-Fuse outperforms its pseudo GT generator EMMA on three clean datasets: MSRS, M3FD, and LLVIP (e.g., \(Q_M\): 1.5051 vs 0.7291 on MSRS), proving that pseudo GT does not limit the model's upper bound performance.

Ablation Study¶

Scenario	Configuration	\(Q_M\uparrow\)	\(Q_G\uparrow\)	SSIM\(\uparrow\)	\(VIF\uparrow\)	\(Q^{AB/F}\uparrow\)	Description
Snow	w/o TDAS	0.5097	0.3951	0.3961	0.3268	0.5012	The fusion quality significantly decreases without TDAS, showing insufficient structural detail restoration
Snow	AMG-Fuse	0.6408	0.4240	0.4302	0.3463	0.5519	Full model
Rain	w/o MGLS	0.6782	0.3910	0.3928	0.3531	0.4982	The structure-sensitive metrics \(Q_G\) and \(Q_M\) exhibit clear degradation without MGLS, with an average reduction of 3.34%
Rain	AMG-Fuse	0.6940	0.4163	0.4080	0.3582	0.5184	Full model
Haze	w/o MCCA	0.5012	0.4178	0.3616	0.3184	0.5217	Degrades to a unidirectional restoration structure without MCCA, limiting cross-modal interaction, with an average reduction of approximately 6.9%
Haze	AMG-Fuse	0.5438	0.4376	0.4083	0.3445	0.5414	Full model

Key Findings¶

MCCA contributes the most: Removing MCCA results in a performance drop of approximately 6.9%, far exceeding that of TDAS and MGLS. This indicates that cross-modal cross-attention is the core of the model; without it, the network degenerates into a unidirectional restoration structure, and the exchange of complementary information between infrared and visible modalities is almost suspended.
TDAS is crucial for severe degradation: In snowy scenarios, removing TDAS causes \(Q_M\) to plunge from 0.6408 to 0.5097 (a 20.5% drop), because snow involves large-area coverage and difficult structural detail restoration, lacking clear "de-degradation" guidance without the degradation-aware constraint.
MGLS annealing mechanism is effective: Ablation reveals that MGLS mainly protects structure-sensitive metrics (\(Q_G\) and \(Q_M\) drop), whereas human perception metrics (\(VIF\) and \(Q_{CB}\)) degrade minimally, corroborating that the primary role of MGLS is to constrain the modal allocation structure rather than the overall appearance.
AMG-Fuse is highly competitive in clean scenes: Although designed for degraded environments, AMG-Fuse still outperforms its pseudo GT generator EMMA on clean datasets, proving that the mask-guided mechanism learns generalized modal interaction principles rather than degradation-specific shortcuts.

Highlights & Insights¶

Mask as a transferable "Modal Fingerprint": The mask \(M\) derived from the fusion equation is essentially a reverse-engineering of the modal allocation strategy of any fusion algorithm—given the fusion result, \(M\) tells you the visible proportion of each pixel. This concept can be transferred to any scenario requiring reverse-engineering of weight allocations from black-box fusion results, such as multi-exposure fusion and multi-focus fusion.
Using the restoration model as a "Cleanliness Detector" instead of a restorer: The ingenuity of TDAS lies in not requiring \(\mathcal{R}\) to actually restore the degraded image, but rather leveraging the property that a "clean input undergoes an approximate identity mapping through the restoration model" to evaluate feature cleanliness. This provides a zero-additional-annotation degradation-aware signal, which can be extended to other tasks requiring assessments of "whether the output is clean".
\(\lambda\) decay is the key stabilizer for pseudo GT methods: The curriculum learning strategy of using pseudo GT to guide convergence early on and yielding to source image supervision later allows the model to enjoy simplified optimization while avoiding overfitting to the bias of pseudo GT. This curriculum-like loss weighting scheme can be generalized to other methods leveraging approximate labels.
Denominator correction in Equation (5): Adding the \(Fuse\) information in the denominator is a simple yet crucial mathematical correction that simultaneously resolves visible brightness bias amplification in adverse weather and numerical instability in night scenes, demonstrating that domain-specific numerical stabilization designs are more effective than general normalization techniques.

Limitations & Future Work¶

High computational overhead: Requires 242.03G FLOPs and 59.74M parameters under a \(224 \times 224\) input, with the main bottleneck being the HTB Histogram Transformer. The authors acknowledge the need for architectural efficiency. Lightweighting directions could explore highly efficient long-range dependency modules (such as the State Space Model Mamba) to replace HTB, or prune the intensity partitioning strategy in HTB.
Insufficient ablation on the choice of pseudo GT generators: The paper solely adopts EMMA as the pseudo GT generator, without comparing the impacts of alternative generators (such as stronger SOTA fusion methods) on final performance. Whether employing a stronger pseudo GT generator can further boost performance or conversely introduce more severe modal bias remains an unexplored question.
Limited adverse weather types: The current scope covers only snow, rain, and haze, without addressing real-world scenes with mixed degradations (e.g., rain + haze, low-light + noise). Under mixed degradations, mask behaviors could be more complex, and degradations might be coupled.
Downstream tasks only validated on detection: Although object detection results are improved, the applicability to other downstream tasks like semantic segmentation or depth estimation remains unverified. Different downstream tasks may define fusion quality differently; whether the current unified framework is optimal deserves further investigation.

vs. Two-stage "restoration-then-fusion" methods: Traditional schemes perform restoration followed by fusion, separating the optimization of both phases and inevitably leading to error accumulation. In contrast, AMG-Fuse embeds restoration as an auxiliary supervision signal into the fusion network via TDAS within a single-stage process, achieving unified optimization of implicit restoration and explicit fusion. This coupled design exhibits prominent advantages under severe degradation.
vs. Text-guided fusion methods (Text-DiFuse, Text-IF): Text-guided approaches use semantic text as an external prior to help the network focus on degraded areas, essentially introducing high-level semantic supervision. The mask in AMG-Fuse, however, is a low-level pixel-level prior derived directly from the physical model of modal contribution, which is closer to the essence of the fusion problem. The two could potentially be combined—using semantic text to guide the precise localization of degraded areas within the mask.
vs. AWFusion (all-weather IVIF): AWFusion introduces various physical model priors into a student network through knowledge distillation, but essentially still follows the "restoration-then-fusion" logic. AMG-Fuse does not rely on physical degradation models but instead learns modal allocation principles from data via masks, potentially exhibiting stronger generalization capacity to unseen degradation types.

Rating¶

Novelty: 4 stars. Deriving the modal contribution mask from the fusion formulation and constructing a complete feature extraction and learning strategy suite around it is highly novel and self-consistent. However, the mask-based attention itself is not entirely new; the increment lies in binding it to the physical constraints of the fusion task.
Experimental Thoroughness: 4 stars. Covers three types of adverse weather, real-world data, clean scenarios, and downstream detection, comparing against 7 SOTAs. Ablations cover three core components and further verify that the pseudo GT does not limit the performance upper bound. However, it lacks experiments on mixed degradations and comparisons of different pseudo GT generators.
Writing Quality: 4 stars. The equation derivation is step-by-step (with a clear logical chain from Eq. (1) to (2) and (3-5)), the problem introduction is natural (double-edged sword of pseudo GT), and Figures 1 and 2 are intuitive. The ablation section is slightly scattered across scenarios, with each component only ablated under one type of weather, which hampers cross-comparison of the relative importance of each component.
Value: 4 stars. The mask decoupling approach is highly generalizable and can be extended to other multi-modal fusion scenarios. The code is open-source, providing direct reference value for image fusion research and applications under adverse weather. However, the substantial computational overhead places actual deployment at a distance.