MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction

Conference: ECCV2026
arXiv: 2604.01958
Code: https://github.com/ixilai/MAVFusion
Area: Video Understanding
Keywords: Infrared and visible video fusion, motion-aware, sparse attention, motion-static decoupling, optical flow guidance

TL;DR

MAVFusion proposes a motion-aware sparse interaction mechanism, which decouples the cross-frame cross-modal interaction in infrared-visible video fusion into two paths based on optical flow-guided motion region detection: "weak interaction for static backgrounds + sparse global attention for dynamic regions." While achieving SOTA fusion quality across three benchmarks, it reduces computational complexity to approximately 5% of existing video fusion methods.

Background & Motivation

Infrared and visible image fusion combines the high saliency of thermal targets (such as high-temperature pedestrians or vehicles) from infrared with the rich texture details of visible light, playing an important role in scenarios such as security monitoring, autonomous driving, and night reconnaissance. However, most existing fusion methods are designed for single-frame images, ignoring inter-frame temporal dependencies, which leads to flickering and ghosting artifacts when directly applied to video scenarios. Although existing video fusion methods improve temporal consistency by introducing cross-frame interactions (such as 3D convolutions or global attention), they perform these dense calculations across all regions of the entire frame at a high computational cost. In reality, most video regions are static backgrounds (roads, walls, grass) where simple and stable texture preservation is sufficient, making strong interaction a waste of computational resources.

The Key Challenge of this "unbiased global modeling" is that videos naturally provide motion cues. Optical flow can precisely locate where substantial changes are occurring (e.g., walking people, moving vehicles, camera shaking), and only these regions truly require complex cross-frame cross-modal interactions to eliminate ghosting and capture temporally salient targets. In contrast, static background regions occupy most of the frame. Applying heavy interactions in these areas may introduce infrared noise or smooth out clear visible textures, degrading the overall fusion quality.

The Key Insight of this paper is to utilize optical flow as a guide, decoupling the fusion interaction into dynamic and static regions, allocating high computational power only to key motion areas. Core Idea: Design an optical flow-guided motion-aware sparse interaction framework. By utilizing optical flow to generate a motion mask, video frames are divided into dynamic regions (processed via Top-K sparse attention for global cross-modal interaction) and static regions (processed via depthwise separable convolutions for lightweight local detail preservation) paths. This is paired with a cross-modal residual optical flow alignment module to eliminate inter-frame ghosting, achieving SOTA fusion quality while reducing computational cost by 20 times.

Method

Overall Architecture

MAVFusion is a four-stage pipeline: input of three adjacent frames from infrared and visible modalities \(\to\) shallow feature extraction + optical flow estimation \(\to\) Motion-Aware Feature Alignment Module (MAFM) for cross-frame alignment and temporal aggregation \(\to\) Motion-Guided Dual-path Interaction Module (MDIM) for separate processing of dynamic/static regions \(\to\) image reconstruction decoder for outputting the fused result.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Infrared + Visible<br/>Three Adjacent Frames Input"] --> B["Shallow Feature Extraction<br/>+ SEA-RAFT Optical Flow Estimation"]
    B --> C["Motion-Aware Feature Alignment Module MAFM<br/>Coarse Alignment → Cross-Modal Residual Refinement → Motion-Gated Temporal Aggregation"]
    C --> D{"Optical Flow Magnitude Generates<br/>Motion Mask M"}
    D -->|Dynamic Region<br/>Top-K Active Patches| E["Dynamic Branch: Sparse Global Attention<br/>Q=Active Patches, KV=Globally Pooled Tokens"]
    D -->|Static Region| F["Static Branch:<br/>Depthwise Separable Convolution for Local Modeling"]
    E --> G["Spatially Gated Adaptive Fusion<br/>Mask Weighting + Smoothing Convolution"]
    F --> G
    G --> H["Image Reconstruction Decoder<br/>→ Fused Frame"]

Key Designs

1. Motion-Aware Feature Alignment Module (MAFM): Coarse-to-Fine Cross-Modal Temporal Alignment

Adjacent frame object motion can cause direct fusion to produce ghosting and blurriness. MAFM first estimates the inter-frame optical flow at a downsampled resolution using a frozen SEA-RAFT to perform coarse alignment. Then, taking the visible features as spatial anchors, it concatenates the coarsely aligned three-frame features, the original optical flow, and the anchor features into a lightweight residual prediction network (composed of depthwise separable convolutions) to output a residual optical flow \(\Delta\phi\) to correct the spatial misalignment between modalities. The refined central frame features and the aligned adjacent frame features are temporally aggregated via a learnable Softmax weighting. However, the aggregated features do not directly replace the entire map; MAFM uses a motion mask calculated from the optical flow magnitude as a soft gate: dynamic regions are injected with multi-frame aggregated information to eliminate ghosting, while static regions rely mainly on single-frame features to maintain background stability. This pixel-wise soft gating mechanism avoids unnecessary cross-frame blending and serves as the foundation for subsequent motion-static decoupled interaction.

2. Motion-Guided Dual-path Interaction Module (MDIM): Motion-Guided Sparse Attention + Static Lightweight Preservation

The core of MDIM is to distinguish between two processing paths using the motion mask \(M\). The dynamic branch first splits the feature map into \(p \times p\) non-overlapping patches and performs adaptive average pooling of the motion mask on each patch to obtain a saliency score \(S\). The active patch index set \(\Omega\) corresponding to the Top-\(\tau\) (\(\tau=0.3\)) scores is retained. The features of these active patches serve as the Query, while the globally pooled tokens of all patches serve as the Key/Value, forming an "asymmetric sparse attention": the complexity is reduced from \(\mathcal{O}(N^2)\) to \(\mathcal{O}(k\cdot N)\) (where \(k\) is the number of active patches, typically \(k \approx 0.3N\)). Note that relative ranking (Top-K instead of a hard thresholding strategy) is used here, which robustly captures the relative foreground motion even in scenarios with large camera shake. The static branch simply utilizes a \(3\times 3\) depthwise separable convolution for local modeling to preserve background textures at an extremely low cost.

3. Spatially Gated Adaptive Feature Reconstruction: Avoiding Ghosting Artifacts at Dynamic-Static Boundaries

The sparse attention features output by the dynamic branch are weighted by the saliency weight \(W\) and then dispersed back to their original spatial positions via index copying, while the static branch retains the full-image features. Both are blended using a spatially soft-gated mixture governed by the upsampled motion mask \(M\), and then passed through a \(3\times 3\) smoothing convolution to eliminate splicing artifacts caused by patch partitioning, ensuring smooth semantic transition between dynamically enhanced regions and statically preserved regions.

Loss & Training

The total loss consists of a spatial loss \(\mathcal{L}_{spatial}\) (pixel similarity + SSIM) to maintain single-frame fusion quality, and a temporal consistency loss \(\mathcal{L}_{temp}\) to constrain the optical flow-guided difference between adjacent frames, balanced with a weight parameter \(\gamma\).

Key Experimental Results

Main Results

Compared with 7 image fusion methods (UP-Fusion, TDFusion, SAGE, GIFNet, RFFusion, UMCFuse, FreeFusion) and 2 video fusion methods (VideoFusion, UniVF) on three infrared-visible video datasets: M3SVD, HDO, and VTMOT.

Dataset	Metric	MAVFusion	Prev. Best (UniVF)	Gain
M3SVD	Q_G↑	0.6897	0.6376	+8.2%
M3SVD	Q_M↑	1.1544	0.5661	+103.9%
M3SVD	Q_AB/F↑	0.7550	0.7049	+7.1%
HDO	Q_G↑	0.6629	0.6125	+8.2%
HDO	Q_M↑	0.9996	0.7655	+30.6%
VTMOT	Q_G↑	0.6325	0.5724	+10.5%
VTMOT	Q_M↑	0.9741	0.8993	+8.3%

For computational efficiency, at a resolution of 640×480, MAVFusion requires only 123.37G FLOPs (6.6% of VideoFusion, 5.7% of UniVF), reaching 14.16 FPS; at 1280×720, it requires 267.88G FLOPs (4.8% of VideoFusion, 3.8% of UniVF). Crucially, when pixels increase by 3 times, the FLOPs only increase by 2.17 times, demonstrating excellent scalability.

Ablation Study

Configuration	Q_G↑	Q_M↑	MS2R↓	Description
Full (Complete Model)	0.6325	0.9741	0.8301	Motion-static decoupling + sparse attention + MAFM
Full-DB (All Dynamic Branch)	0.6276	0.9282	0.9878	Full-image intensive interaction, dispersed attention, background interference
Full-SB (All Static Branch)	0.6260	0.9550	0.9913	No dynamic enhancement, blurry moving targets, degraded MS2R
w/ IM (Inverted Mask)	0.5880	0.7863	0.9842	Intensive interaction in static regions introduces infrared noise, blurry textures
w/o SA (Replace with dense attention)	0.6080	0.8351	0.9662	Dispersed attention distribution, degraded local details
w/o MAFM (Replace with UniVF alignment)	0.5972	0.7820	0.9799	No cross-modal anchor refinement, increased ghosting

Key Findings

• The motion-static decoupling strategy is crucial: The Full-DB experiment indicates that applying global, heavy interaction to the entire map not only wastes computation but also weakens both salient targets and the background during attention competition. The inverted mask (weak interaction for dynamic regions and strong interaction for static regions) results in the most severe performance drop, demonstrating that which interaction mode should be applied to which region is non-interchangeable.
• Sparse attention outperforms dense attention: As shown by w/o SA, replacing sparse attention with dense fully connected attention within patches leads to a decline in quantitative metrics and results in visual detail degradation, indicating that densely covering redundant information tends to smooth out local features.
• The cost of the front-end optical flow is the primary limitation: Although the fusion network itself has extremely low computational complexity, the overhead of the front-end SEA-RAFT optical flow estimation remains a bottleneck (the paper discusses replacing it with lightweight optical flow models to accelerate future work).

Highlights & Insights

• The "Motion-Static Decoupling" concept is highly generalizable: Infrared-visible fusion requires distinct interaction modes for motion and static regions. This observation can be extended to other multi-modal video tasks (such as video depth estimation and video semantic segmentation) where the scene layout follows a "mostly static background + minor foreground motion" distribution.
• Ingenious Design of Asymmetric Sparse Attention: Utilizing all globally pooled patch tokens as Key/Value and only active patches as Query preserves the global receptive field (allowing active patches to capture context from all locations) while reducing the complexity of the Query side to \(O(kN)\), which is more flexible than using local windows.
• Top-K Relative Ranking Outperforms Fixed Thresholds: The relative ranking strategy is robust to camera shake and large motion scenarios, preventing the entire frame from being classified as a "dynamic region" when global motion increases, which would degrade sparse attention back to dense attention.

Limitations & Future Work

• The authors pointed out that severe sensor noise or extreme degradation (such as complete absence of light in the visible channel) can still damage modal feature extraction, making it difficult for the model to completely suppress degradation artifacts.
• Currently, the optical flow model uses a frozen SEA-RAFT, and its extra overhead accounts for the major part of the execution time. Future work can explore end-to-end joint training of a lightweight motion estimator, or directly replace optical flow with simple motion detection to further accelerate inference.
• The motion-static decoupling assumes that most areas in a scene are static. In densely moving scenes (e.g., sports events, crowded streets), the performance might decline, and the Top-K retention ratio \(\tau=0.3\) may need adaptive tuning.

Related Work & Insights

• vs UniVF (NeurIPS 2025): UniVF enhances temporal consistency via multi-frame learning and optical flow feature registration, but still performs global modeling across the entire image, incurring about 18 times the FLOPs of MAVFusion. MAVFusion's motion-static decoupling directly eliminates redundant interactions in static backgrounds.
• vs VideoFusion (CVPR 2026): Also designed for video fusion, it employs a hierarchical fusion strategy to reduce inter-frame feature divergence, but its computational cost is 1874G FLOPs (at 640×480), which is approximately 15 times that of MAVFusion.
• vs Image Fusion Methods (UP-Fusion, TDFusion, etc.): These methods ignore temporal modeling, leading to ghosting and flickering in video scenes, which MAVFusion's MAFM module and temporal loss explicitly resolve.

Rating

• Novelty: ⭐⭐⭐⭐ Translating the observation of "mostly static video regions" into a motion-static decoupled framework, and the application of sparse asymmetric attention in fusion tasks is highly novel.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated on three datasets under eight metrics, with extensive ablation studies (5 configurations), downstream detection task validation, and a comprehensive computational efficiency comparison.
• Writing Quality: ⭐⭐⭐⭐ The motivation is clear, and the logical chain between design choices is complete, though some mathematical formulations are slightly redundant.
• Value: ⭐⭐⭐⭐⭐ Provides an extremely practical and efficient solution for infrared-visible video fusion, with a motion-static decoupling paradigm that can easily be extended to other multi-modal video tasks.

Related Papers

• [CVPR 2026] InterRVOS: Interaction-Aware Referring Video Object Segmentation
• [ECCV 2026] Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
• [AAAI 2026] Explicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction
• [AAAI 2026] PlugTrack: Multi-Perceptive Motion Analysis for Adaptive Fusion in Multi-Object Tracking
• [ICCV 2025] Sparse-Dense Side-Tuner for Efficient Video Temporal Grounding