Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction¶

Conference: ECCV 2026
arXiv: 2606.26812
Code: https://github.com/ixilai/AMG-Fuse
Area: Multimodal VLM
Keywords: Multi-modality Image Fusion, Infrared-Visible Image Fusion, Adverse Weather Image Restoration, Mask-Guided Learning, Cross-Modal Attention

TL;DR¶

AMG-Fuse proposes a mask-guided multi-modality image fusion framework under adverse weather. By decoupling modal contribution masks from the "Pseudo Ground Truth", and combining them with the Mask-guided Feature Extraction Module (MFEM) featuring Multi-modal Cross-modal Cross-Attention (MCCA), a Mask-Guided Learning Strategy (MGLS), and a Task-coupled Degradation-Aware Learning Strategy (TDAS), the framework simultaneously achieves feature restoration and cross-modal interaction within a unified network. It comprehensively outperforms previous state-of-the-art (SOTA) methods under three adverse weather conditions (snow, rain, haze) and real-world scenes, achieving the best mAP in downstream object detection tasks.

Background & Motivation¶

Multi-modality image fusion (MMIF) generates richer scene representations by integrating complementary information from various modalities. For instance, in infrared-visible image fusion (IVIF), visible images capture textural details while infrared images highlight salient targets via thermal radiation; their fusion benefits downstream tasks such as object detection and semantic segmentation. Existing IVIF methods primarily operate under ideal scenarios, but real-world adverse weather conditions (snow, rain, haze) cause severe image degradation, destroying feature representations and making it difficult for the fusion network to simultaneously achieve the dual goals of "feature restoration" and "cross-modal complementation."

Existing solutions dealing with adverse weather suffer from inherent limitations. The first category is the "restoration then fusion" two-stage paradigm, which processes each modality using independent restoration networks before feeding them into a fusion network. This introduces two issues: the restoration stage focuses on intra-modal reconstruction while the fusion stage focuses on inter-modal complementation, where inconsistent optimization objectives lead to unstable training; and artifacts generated during the restoration stage propagate into the fusion stage, accumulating errors. The second category employs "Pseudo Ground Truth" (Pseudo GT) as supervision—namely, using fusion results generated by existing fusion methods on clean image pairs as training targets. This approach simplifies the optimization objective to \(\theta^* = \arg\min_\theta \|f(VI, IR; \theta) - GT_{Pse}\|_1\), reducing optimization complexity and helping preserve global structure and details. However, the pseudo ground truth itself carries information loss and modal bias—the network easily overfits to the static pixel distribution of the pseudo ground truth instead of learning a dynamic modal allocation mechanism, which may even lead to the loss of key infrared thermal target information in clean scenes.

Key Challenge: While pseudo ground truth simplifies training, it inclines the network to mechanically replicate surface features, thereby neglecting the dynamic mining of cross-modal complementary cues. Core Idea: Deriving a modal contribution mask \(M\) (satisfying \(Fuse = M \times VI + (1-M) \times IR + \varepsilon\)) from the mapping relationship between the pseudo ground truth and source images. Using this mask as an explicit external prior to model cross-modal interactions forces the network to learn the allocation rule of "how much each modality contributes to the pseudo ground truth" rather than simply copying the pixel values of the pseudo ground truth.

Method¶

Overall Architecture¶

The input of AMG-Fuse is a pair of degraded infrared image \(IR\) and visible image \(VI\), and the output is a de-degraded, thoroughly fused image. The entire network inherits an end-to-end single-stage structure. The core idea is to embed the modal contribution mask into the feature extraction and interaction processes, enabling the network to dynamically allocate fusion weights for each modality while restoring degraded features.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["IR 红外 + VI 可见光<br/>（退化输入）"] --> B["卷积 + 残差块<br/>浅层特征提取"]
    B --> C["残差块细化<br/>模态专属特征"]
    B --> D["HTB 直方图 Transformer<br/>强度分区自注意力"]
    C --> E["掩码引导特征提取 MFEM<br/>掩码构建 → MCCA 交叉注意力"]
    D --> E
    E --> F["融合图像输出"]

Specific workflow: First, shallow multimodal features are extracted using convolutions and residual blocks, and are combined to generate initial fused features. Subsequently, modality-specific features are further refined via residual blocks, while fused features are fed into a Histogram Transformer Block (HTB). HTB partitions spatial features according to pixel intensity and performs self-attention within each partition, effectively capturing similar degradation patterns across long distances (such as rain streaks). Finally, the refined multimodal features and the HTB-enhanced fused features enter the Mask-guided Feature Extraction Module (MFEM) to perform mask construction and cross-modal cross-attention (MCCA), producing the final fused image.

During training, AMG-Fuse utilizes pseudo ground truth \(GT_{Pse}\) (generated by EMMA on clean image pairs) as auxiliary supervision, optimized jointly with MGLS, TDAS, gradient loss, and color consistency loss. Among these, the weight coefficient \(\lambda\) of MGLS gradually decays as training epochs increase, allowing the pseudo ground truth to stabilize training in the early stages and yield to the supervision of clean source images in the later stages.

Key Designs¶

1. Modal Contribution Mask Construction: Decoupling Modal Allocation Weights from Fusion Formulas

The core objective of IVIF is to suppress cross-modal redundancy and extract complementary information. Ignoring feature enhancement and dimensionality reduction, the fusion result can be expressed as \(Fuse = M \times VI + (1-M) \times IR + \varepsilon\), where \(M\) is the modal weight allocation mask and \(\varepsilon\) is the error and noise term. When \(Fuse\), \(VI\), and \(IR\) are known, directly solving \(M = \frac{Fuse - IR}{VI - IR} + \varepsilon_M\) under adverse weather leads to severe problems: under hazy/snowy days, the visible image brightness may be overexposed and the infrared image contrast is low, leading the denominator \(VI - IR\) to be dominated by degradation factors rather than semantic content, meaning the mask fails to reflect the true modal distribution; in night scenes, \(VI - IR\) is close to zero or negative over large areas, causing numerical instability.

The key modification proposed is to introduce the \(Fuse\) information into the denominator, rewriting it as \(M = \frac{Fuse - IR}{VI - IR + Fuse} + \varepsilon_M\). This operation achieves two goals: the optimized and de-degraded \(Fuse\) exhibits a more stable structure, effectively suppressing the erroneous amplification of masks caused by visible brightness bias; it also avoids numerical instability when the denominator approaches zero. Performing feature decomposition on the pseudo ground truth using this mask yields the visible contribution map \(F_{VI} = M_{Pse} \times VI_C\) and the infrared contribution map \(F_{IR} = (1-M_{Pse}) \times IR_C\), intuitively showcasing the allocation strategies of different fusion algorithms for each modal information.

2. Mask-Guided Feature Extraction Module (MFEM) and Cross-Modal Cross-Attention (MCCA)

Direct training on pseudo ground truths causes the network to replicate inherent biases of the pseudo ground truth, emphasizing the reproduction of surface features while omitting effective cross-modal interactions. The design of MFEM follows the synthetic paradigm of the equation \(Fuse = M \times VI + (1-M) \times IR\): first, multimodal features are fed into a Residual Smooth Block (RSB) for refinement, and fused features are enhanced via HTB; then, the mask \(M\) is calculated based on both; finally, the network learns multimodal interaction patterns in the pseudo ground truth via the MCCA module.

MCCA adopts a cross-attention mechanism: multimodal features act as Queries guided by mask weighting, while fused features act as Keys and Values. Specifically, the mask weights the visible Query \(Q_{VI}\) and infrared Query \(Q_{IR}\) respectively, guiding the network to focus on the salient features in each modality. Query and Key-Value branches introduce depthwise separable convolutions to expand the local receptive field. Performing cross-attention between the mask-weighted Query and the Key/Value of the fused features accomplishes the decoupling and reorganization of multimodal features in the fusion space—essentially enabling the network to learn "how much visible and infrared information should be extracted at this position during fusion" rather than blindly replicating the pseudo ground truth.

3. Mask-Guided Learning Strategy (MGLS): Supervising Dynamic Fusion with Modal Distributions from Pseudo Ground Truth

The core idea of MGLS is to use the modal allocation pattern within the pseudo ground truth to constrain the network. It first computes the mask \(M_{Pse}\) using Eq. (5) based on the pseudo ground truth \(GT_{Pse}\) and clean source images \(VI_C, IR_C\), and then decouples the visible allocation map \(F_{VI}\) and infrared allocation map \(F_{IR}\) from the pseudo ground truth according to \(F_{VI} = M_{Pse} \times VI_C, F_{IR} = (1-M_{Pse}) \times IR_C\). Subsequently, an L1 loss is applied to the corresponding modal features \(\hat{F}_{VI}\) and \(\hat{F}_{IR}\) output by the network:

\[\mathcal{L}_{MGLS} = \frac{1}{HW} (\|\hat{F}_{VI} - F_{VI}\|_1 + \|\hat{F}_{IR} - F_{IR}\|_1)\]

This forces the network to learn the distribution of "how much each modality contributes to the pseudo ground truth" instead of simply fitting the overall appearance of the pseudo ground truth. Crucially, the weight coefficient \(\lambda\) of MGLS decays during training: \(\lambda\) is larger in the early stages to stabilize training and accelerate convergence using the pseudo ground truth; it gradually decreases in the later stages, allowing supervision from clean source images to dominate the optimization and preventing the network from being locked into the biases of the pseudo ground truth.

4. Task-Coupled Degradation-Aware Learning Strategy (TDAS): Using a Restoration Model as a "Cleanliness Detector"

The design motivation of TDAS stems from a key observation: the mask \(M_{Deg}\) under degraded scenarios can effectively capture the distribution of degraded regions—for example, in rainy scenes, \(M_{Deg}\) successfully suppresses most rain streaks. In the equation \(Fuse = M_{Deg} \times VI_{Deg} + (1-M_{Deg}) \times IR_{Deg} + \varepsilon\), since \(Fuse\) is a degradation-free image, \(VI_F = M_{Deg} \times VI_{Deg}\) should theoretically represent the degradation-free visible component.

Based on this, TDAS introduces a pre-trained restoration model \(\mathcal{R}(\cdot)\) as a "cleanliness detector": if \(VI_F\) is sufficiently restored, the output of the restoration model \(\mathcal{R}(VI_F)\) on it should approximate an identity map (since the input itself is already clean). The TDAS loss is defined as:

\[\mathcal{L}_{TDAS} = \frac{1}{H \times W} \|VI_F - \mathcal{R}(VI_F)\|_1\]

When the fusion network successfully restores clean features, \(VI_F\) is close to \(\mathcal{R}(VI_F)\), and the loss is small. Conversely, if the fusion network fails to remove degradation (e.g., rain streaks remain), the restoration model performs deraining, causing the output to change and the loss to increase. This is equivalent to using the restoration task as an auxiliary supervision signal, enhancing the network's degradation awareness and guiding it to prioritize clear and salient regions. It is complementary to MGLS: TDAS focuses on "whether features are clean," while MGLS focuses on "how modalities are allocated."

Loss & Training¶

In addition to MGLS and TDAS, AMG-Fuse introduces two source image supervision losses to enhance the model's ability to capture multimodal distributions. Gradient loss preserves structural details: \(\mathcal{L}_{grad} = \frac{1}{HW} \|\nabla Fuse - \max(|\nabla VI_C|, |\nabla IR_C|)\|_1\), requiring that the gradient intensity of the fused image is not lower than the maximum of the two source images. Color consistency loss maintains color distribution: \(\mathcal{L}_{color} = \frac{1}{HW} \|F_{CbCr}(Fuse) - F_{CbCr}(VI_C)\|_1\), which constrains the color components of the fusion results to be consistent with the visible image after converting the image from RGB to the CbCr color space.

The total loss is defined as \(\mathcal{L}_{total} = \lambda \times \mathcal{L}_{MGLS} + \mathcal{L}_{TDAS} + \mathcal{L}_{color} + \mathcal{L}_{grad}\), where \(\lambda\) is the decay coefficient that decreases progressively with training epochs; the weights of the remaining three terms are all set to 1, requiring no fine-tuning. The training set comprises 1,000 images each for snow, rain, and haze from the AWMM-100k dataset, cropped into \(168 \times 168\) patches. Optimization is done via the Adam optimizer (initial learning rate of \(1\times 10^{-3}\), batch size of 2) on four RTX 3090 GPUs for 200 epochs.

Key Experimental Results¶

Main Results¶

Under three adverse weather conditions (snow, rain, haze), comparisons are conducted against 7 baseline methods (LRRNet, Text-DiFuse, EMMA, Text-IF, GIFNet, SAGE, AWFusion). Except for AWFusion, which processes degradation in an end-to-end manner, all other methods employ AdaIR as a pre-restoration step. AMG-Fuse ranks in the top two on the vast majority of metrics.

Scene	Method	\(Q_M\uparrow\)	\(Q_G\uparrow\)	SSIM\(\uparrow\)	\(VIF\uparrow\)	\(Q^{AB/F}\uparrow\)
Snow	EMMA (CVPR 24)	0.5353	0.4235	0.4096	0.3252	0.5470
Snow	Text-IF (CVPR 24)	0.5942	0.4199	0.4298	0.3300	0.5419
Snow	AWFusion (INFFus 26)	0.5535	0.3552	0.3734	0.3024	0.4937
Snow	AMG-Fuse	0.6408	0.4240	0.4302	0.3463	0.5519
Rain	EMMA (CVPR 24)	0.5871	0.3745	0.3700	0.3384	0.4866
Rain	Text-IF (CVPR 24)	0.6375	0.4085	0.4046	0.3489	0.5149
Rain	AMG-Fuse	0.6940	0.4163	0.4080	0.3582	0.5184
Haze	EMMA (CVPR 24)	0.4149	0.3553	0.3296	0.2559	0.4617
Haze	AWFusion (INFFus 26)	0.5471	0.4283	0.3901	0.3433	0.5322
Haze	AMG-Fuse	0.5438	0.4376	0.4083	0.3445	0.5414

Under real-world scenarios (real hazy data), AMG-Fuse ranks in the top two across all seven metrics (\(Q_{MI}\): 0.7234, \(Q_G\): 0.5114, \(Q_M\): 1.1022, \(VIF\): 0.4373, SSIM: 0.5202, \(Q^{AB/F}\): 0.5957).

Downstream tasks: For object detection on the M3FD dataset using YOLOv7, AMG-Fuse achieves the best [email protected]=0.843 and [email protected]:0.95=0.541, ranking in the top two in 5 out of 6 categories.

Outperforming the Pseudo GT Generator: AMG-Fuse outperforms its pseudo GT generator EMMA on three clean datasets, namely MSRS, M3FD, and LLVIP (e.g., \(Q_M\): 1.5051 vs 0.7291 on MSRS), proving that the pseudo GT does not limit the model's upper bound.

Ablation Study¶

Scene	Config.	\(Q_M\uparrow\)	\(Q_G\uparrow\)	SSIM\(\uparrow\)	\(VIF\uparrow\)	\(Q^{AB/F}\uparrow\)	Description
Snow	w/o TDAS	0.5097	0.3951	0.3961	0.3268	0.5012	Removing TDAS significantly degrades fusion quality, and structural detail restoration is insufficient
Snow	AMG-Fuse	0.6408	0.4240	0.4302	0.3463	0.5519	Full model
Rain	w/o MGLS	0.6782	0.3910	0.3928	0.3531	0.4982	Removing MGLS leads to a significant decrease in structure-sensitive metrics \(Q_G\) and \(Q_M\), with an average decline of 3.34%
Rain	AMG-Fuse	0.6940	0.4163	0.4080	0.3582	0.5184	Full model
Haze	w/o MCCA	0.5012	0.4178	0.3616	0.3184	0.5217	Removing MCCA degrades the model into a unidirectional restoration structure, restricting cross-modal interactions, with an average decline of approximately 6.9%
Haze	AMG-Fuse	0.5438	0.4376	0.4083	0.3445	0.5414	Full model

Key Findings¶

MCCA contributes the most: Removing MCCA results in an average performance drop of approximately 6.9%, far exceeding TDAS and MGLS. This indicates that cross-modal cross-attention is the core of the model—without it, the network degrades into a unidirectional restoration structure, and the complementary information exchange between the infrared and visible modalities is almost entirely interrupted.
TDAS is particularly critical for severe degradation: In the snowy scene, removing TDAS causes \(Q_M\) to plummet from 0.6408 to 0.5097 (a 20.5% drop). Because snow coverage is extensive and structural details are difficult to recover, the network lacks explicit "de-degradation" guidance without degradation-aware constraints.
MGLS annealing mechanism is effective: Ablation results show that MGLS primarily protects structure-sensitive metrics (\(Q_G\) and \(Q_M\) drop), while human perception metrics (\(VIF\), \(Q_{CB}\)) exhibit smaller declines. This supports that the core role of MGLS is to constrain the modal allocation structure rather than the overall appearance.
AMG-Fuse is equally competitive in clean scenes: Although designed for degraded scenarios, AMG-Fuse still outperforms its pseudo ground truth generator EMMA on clean datasets, proving that the mask-guided mechanism learns general modal interaction rules rather than degradation-specific shortcuts.

Highlights & Insights¶

Mask as a transferable "modal fingerprint": The mask \(M\) derived from the fusion formula is essentially a reverse engineering of the modal allocation strategy of any fusion algorithm—given the fusion result, \(M\) tells you the proportion of each pixel originating from the visible light. This concept can be transferred to any scenario that "reverses weight allocation from black-box fusion results," such as multi-exposure fusion and multi-focus fusion.
Using a restoration model as a "cleanliness detector" rather than a restorer: The ingenuity of TDAS lies in not requiring \(\mathcal{R}\) to actually restore the degraded image, but rather leveraging the property that "clean inputs undergo an approximate identity mapping through the restoration model" to evaluate feature cleanliness. This provides a zero-extra-annotation degradation-aware signal, and a similar idea could be applied to other tasks that require determining "whether the output is clean."
\(\lambda\) decay as a key stabilizer for pseudo ground-truth methods: The strategy where pseudo ground truths guide convergence in the early stage and yield to source image supervision in the later stage allows the model to enjoy the benefits of simplified optimization while avoiding overfitting to pseudo GT biases. This curriculum-learning-style loss weighting can be generalized to other methods utilizing approximate labels.
Denominator modification in Eq. (5): Adding \(Fuse\) information to the denominator is a simple yet crucial mathematical correction that simultaneously solves both the brightness bias amplification in adverse weather and the numerical instability in night scenes. This demonstrates that numerical stabilization designs tailored to task characteristics are more effective than general normalization techniques.

Limitations & Future Work¶

Heavy computational overhead: It requires 242.03G FLOPs and 59.74M parameters under a \(224 \times 224\) input, with the main bottleneck being the HTB (Histogram Transformer Block). The authors acknowledge the need for efficiency improvements. Lightweight directions could consider replacing HTB with more efficient long-range dependency modules (such as the State Space Model, Mamba) or pruning the intensity partitioning strategy of HTB.
Insufficient ablation on the choice of pseudo ground truth generator: The paper only uses EMMA as the pseudo GT generator, failing to compare the impact of different generators (such as stronger SOTA fusion methods) on the final performance. Whether a stronger pseudo GT generator can further improve performance or instead introduce more severe modal bias remains an open question worth exploring.
Limited adverse weather types: Currently, the model only covers three types of degradation (snow, rain, haze), leaving hybrid degradations in real-world scenarios (e.g., rain + haze, low-light + noise) unaddressed. Mask behaviors under hybrid degradations might be more complex, and couplings could exist between different degradations.
Downstream tasks only validated on detection: Though object detection performance is improved, the adaptability to other downstream tasks, such as semantic segmentation and depth estimation, remains unverified. Different downstream tasks may define fusion quality differently; whether the current unified framework is optimal warrants further research.

vs. Two-stage "restoration then fusion" methods: Traditional schemes perform restoration before fusion, causing a disjoint optimization objective between the two stages and making error accumulation inevitable. Within a single stage, AMG-Fuse embeds restoration as an auxiliary supervision signal into the fusion network via TDAS, achieving unified optimization of implicit restoration and explicit fusion. Experiments prove this coupled design offers significant advantages under severe degradation.
vs. Text-guided fusion methods (Text-DiFuse, Text-IF): Text-guided methods utilize semantic text as an external prior to help the network focus on degraded regions, essentially introducing high-level semantic supervision. Conversely, the mask in AMG-Fuse is a low-level pixel-level prior derived directly from the physical model of modal contributions, which aligns closer to the essence of the fusion problem. The two could potentially be combined—using semantic text to guide the precise localization of degraded areas in the mask.
vs. AWFusion (all-weather IVIF): AWFusion introduces various physical model priors into the student model through knowledge distillation, but essentially retains the "restoration then fusion" line of thought. Rather than relying on physical degradation models, AMG-Fuse learns modal allocation patterns from data via masks, potentially exhibiting stronger generalization capability toward unseen degradation types.

Rating¶

Novelty: 4 stars. Deriving the modal contribution mask from the fusion formula and constructing a complete system of feature extraction and learning strategies around it is both novel and self-consistent. However, the mask-attention mechanism itself is not entirely new; the incremental value lies in binding it to the physical constraints of the fusion task.
Experimental Thoroughness: 4 stars. Covers three types of adverse weather, real-world data, clean scenarios, and downstream detection, comparing against 7 SOTAs. Ablations cover three core components and additionally verify that the pseudo ground truth does not cap the model performance. However, there is a lack of experiments on mixed degradations and comparisons among different pseudo GT generators.
Writing Quality: 4 stars. Formula derivations progress step-by-step (with a clear logical chain from Eq.(1) \(\rightarrow\) Eq.(2) \(\rightarrow\) Eq.(3-5)), the problem introduction is natural (highlighting the dual nature of pseudo ground truths), and Figures 1-2 are intuitive. The ablation section is slightly scattered "by scenario," as each component is ablated under only one weather type, which impedes a direct horizontal comparison of the relative importance of each component.
Value: 4 stars. The mask decoupling idea holds general applicability and can be extended to other multi-modality fusion scenarios. With open-source code, it offers direct reference value for research and application in image fusion under adverse weather. However, the computational overhead is relatively heavy, posing a gap to practical deployment.