FreeMEF: A Flexible Multi-Exposure Fusion Transformer for Arbitrary Number of Frames¶

Conference: ECCV2026
arXiv: 2606.27905
Code: https://github.com/qulishen/FreeMEF
Area: Image Restoration
Keywords: Multi-Exposure Fusion, HDR Imaging, State Space Models, Attention Mechanism, Variable Number of Frames

TL;DR¶

FreeMEF proposes a "There and Back Again" two-stage paradigm: it first recursively aggregates multi-exposure features from an arbitrary number of frames into a global representation using a Recurrent State Space Module, and then restores the reference frame guided by an Extreme-Aware Hybrid Attention. This enables flexible inference on 2/3/5 frames while significantly suppressing ghosting artifacts and enhancing dynamic range.

Background & Motivation¶

Multi-exposure fusion (MEF) reconstructs High Dynamic Range (HDR) images by fusing multiple Low Dynamic Range (LDR) images of the same scene captured under different exposure values, serving as a classic strategy to overcome the limited dynamic range of sensors. Although deep learning-based MEF methods have made remarkable progress recently, they suffer from a fundamental limitation: they assume a fixed exposure strategy, meaning their architectures are designed for a fixed number of frames (e.g., 2, 3, or 5 frames). In real-world deployment, different devices or shooting modes on the same device employ varying numbers of exposure frames. This forces practitioners to train and deploy separate models for each frame count, severely limiting practical efficiency.

A deeper contradiction lies in the mechanism of attention. Existing methods, whether performing self-attention after fusion or using cross-attention between a reference frame and support frames, inherently rely on similarity matching: query vectors are derived from reference frame features, while key-value vectors come from support frame features. However, the core goal of HDR imaging is to recover details in over-exposed or under-exposed regions of the reference frame, where pixel values are clipped or saturated. These regions exhibit extremely low similarity with the normally exposed regions of support frames. This leads to a similarity paradox: the attention mechanism assigns the lowest weights to the regions that need restoration most. Moreover, current parallel fusion strategies tend to fuse misalignment information caused by motion between different frames, making it difficult to completely eliminate ghosting in subsequent processing.

The core idea of this work is to split multi-exposure fusion into a two-step "aggregate-then-guide" process: first, progressively aggregate information from an arbitrary number of frames into a global representation via a recurrent mechanism; second, use this global representation to guide the reference frame restoration in an extreme-aware manner. This "There and Back Again" paradigm naturally decouples the input frame count from architectural constraints. Additionally, by explicitly learning an extreme region map, it bypasses the similarity paradox, allowing the attention mechanism to automatically switch to self-association search in saturated regions.

Method¶

Overall Architecture¶

The overall pipeline of FreeMEF is divided into two main stages. Given a set of exposure frames \(\{\mathbf{I}_t\}_{t=0}^T\) (where \(\mathbf{I}_0\) is the reference frame and the rest are support frames), the first stage employs a Recurrent State Space Module (RSSM) to process the frames sequentially in any order. It progressively aggregates global information into a compact global fused feature \(\mathbf{H}_T\) through deformable alignment and state space recursion. In the second stage, taking the reference frame \(\mathbf{I}_0\) and the global feature \(\mathbf{H}_T\) as inputs, a U-shaped Transformer encoder-decoder architecture resolves the image. At each resolution level, a Global Feature Guided Block (GFGB) is deployed to selectively exploit global context to restore the reference frame. Inside GFGB, two sub-modules are introduced: Extreme-Aware Hybrid Attention (EAHA), which resolves the similarity paradox, and Affine-Injected Feed-Forward Network (AFFN), which handles explicit brightness and contrast adjustments.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Exposure Frames<br/>I0, I1, ..., IT"] --> B["Shared Feature Extraction<br/>FEM"]
    B --> C["RSSM: Frame-by-Frame Recurrent Fusion<br/>Deformable Alignment + ASE<br/>Sigmoid Gated Update"]
    C --> D["Global Fused Feature HT"]
    D --> E["GFGB × Multi-scale<br/>U-Net Encoder-Decoder"]
    E --> F["EAHA: Extreme-Aware<br/>Hybrid Attention"]
    E --> G["AFFN: Affine Modulation<br/>Brightness/Contrast Adjustment"]
    F --> H["Fusion Result I_hat"]
    G --> H

Key Designs¶

1. RSSM: State Space-Based Recurrent Fusion Mechanism

RSSM is the core component that enables FreeMEF to support an arbitrary number of frames. Each input frame \(\mathbf{I}_t\) is first processed by a shared lightweight feature extraction module to obtain shallow features \(\mathbf{F}_t\), which then enter the recurrent unit. The recurrent unit performs two key operations: First, using deformable convolution (DCN) based on the history state \(\mathbf{H}_{t-1}\) and current feature \(\mathbf{F}_t\), it predicts offsets to register the aligned current feature \(\bar{\mathbf{F}}_t\) into the coordinate system of the historical state, thereby resolving spatial misalignment caused by handheld camera shake or scene motion. Second, the aligned features are fed into the Attention State Space Equation (ASE). Unlike the fixed output matrix \(\mathbf{C}\) in standard SSMs which restricts the receptive field to the causal scanning direction, ASE dynamically selects semantic prototypes for each pixel from a learnable prompt pool, expanding the output matrix to \(\mathbf{C}+\mathbf{P}\). This introduces non-causal global context while maintaining linear complexity. After gated activation, the output of ASE is fused with the historical state via a sigmoid gating mask \(\mathbf{G}_t\): \(\mathbf{H}_t = \mathbf{H}_{t-1} + \mathbf{G}_t \odot (\text{update candidate} - \mathbf{H}_{t-1})\). This recurrent architecture naturally accommodates arbitrary frame counts—trained on 3 frames, it can seamlessly switch to 2 or 5 frames during inference without any architectural changes.

2. EAHA: Extreme-Aware Hybrid Attention to Resolve the Similarity Paradox

This is the most ingenious contribution of the paper. Since the reference frame contains extreme numerical values and degraded structures in over-/under-exposed regions, directly generating query vectors \(\mathbf{Q}_{base}\) from the reference frame features to match the global feature \(\mathbf{K}\) yields low similarity—meaning attention weights align least with the regions that need restoration the most. EAHA resolves this by introducing a second query path \(\mathbf{Q}_{ref}\), which is directly generated from the global fused feature \(\mathbf{H}_T\) and acts as a backup search vector for self-association within the global representation. Crucially, an extreme region map \(\mathbf{E}=\sigma(\text{depth-wise separable convolution}(\mathbf{F}_0))\) is utilized to dynamically control the pixel-wise blending ratio: \(\mathbf{Q}_{hybrid} = (1-\mathbf{E})\odot\mathbf{Q}_{base} + \mathbf{E}\odot\mathbf{Q}_{ref}\). In normally exposed regions (\(\mathbf{E}\approx 0\)), the model queries using the reference frame content; in saturated regions (\(\mathbf{E}\approx 1\)), it automatically switches to correlation search within the global features. This "soft switching" ensures that the attention mechanism finds meaningful matches across all regions. Moreover, when attention is computed along the channel dimension, the calculation of \(\mathbf{K}^\top\mathbf{Q}_{hybrid}\) is equivalent to a weighted sum of self-attention and cross-attention, introducing no extra computational overhead.

3. AFFN: Affine-Injected Feed-Forward Network for Explicit Brightness Regulation

Standard FFNs lack the capability to perceive global exposure variations. The intuition behind AFFN is that LDR-to-HDR restoration is not simply a matter of feature addition; it requires explicit correction of brightness statistics (scale and shift). It extracts a global descriptor from the global fused feature \(\mathbf{H}_T\)—first applying global average pooling and a convolution, then splitting into scale parameter \(\gamma\) and shift parameter \(\beta\)—to perform an affine transformation on input features: \(\mathbf{X} \odot (1+\gamma) + \beta\), before feeding them into the gated linear unit. This mechanism enables the model to explicitly regulate local contrast and color shift based on global high dynamic range information, resulting in more natural brightness and fewer color distortions in final visual results.

Loss & Training¶

The network is trained using the standard \(L_1\) loss. The optimizer is Adam (\(\beta_1=0.9,\beta_2=0.999\)), with an initial learning rate of \(2\times 10^{-4}\), employing a cosine annealing warm restart schedule for a total of 300k iterations.

Key Experimental Results¶

Main Results¶

Dataset	Metric	FreeMEF	Prev. SOTA (AFUNet)	Gain
Kalantari et al.	PSNR↑ / SSIM↑ / LPIPS↓	28.418 / 0.948 / 0.081	27.226 / 0.925 / 0.091	+1.19dB PSNR
Real-HDRV	PSNR↑ / SSIM↑ / LPIPS↓	26.077 / 0.940 / 0.074	25.562 / 0.924 / 0.089 (Restormer)	+0.52dB PSNR

Cross-dataset generalization (SICE dataset, trained on Kalantari, directly tested with different frame counts):

Testing Frames	Metric	FreeMEF	Prev. SOTA	Gain
2 frames	PSNR↑ / SSIM↑	17.087 / 0.731	15.339 / 0.690	+1.75dB
3 frames	PSNR↑ / SSIM↑	19.326 / 0.774	18.724 / 0.768	+0.60dB
5 frames	PSNR↑ / SSIM↑	22.269 / 0.851	21.153 / 0.842	+1.12dB

FreeMEF has 8.9M parameters and 41.5G FLOPs, saving 56% of computation compared to HDR-Transformer.

Ablation Study¶

Configuration	PSNR	SSIM	Description
Full FreeMEF	28.418	0.948	Complete model
RSSM → CNN embedding layer	27.082	0.920	Removing RSSM drops PSNR by 1.34dB, highlighting the value of recurrent fusion.
EAHA → MDTA	27.565	0.937	Removing EAHA drops PSNR by 0.85dB, showing extreme-aware hybrid attention is crucial.
AFFN → GDFN	27.982	0.937	Removing AFFN drops PSNR by 0.44dB, indicating significant contribution of affine modulation.

Key Findings¶

RSSM makes the largest contribution (+1.34dB), showing that progressive recurrent fusion significantly outperforms parallel processing with simple CNN embeddings.
Fusion order experiments reveal that placing dark frames (long exposures) at the end of the fusion process helps optimize the final image quality, whereas placing bright frames at the end introduces more noise.
The method exhibits a unique advantage in cross-dataset generalization: a performance gain of 1.75dB in 2-frame scenarios, because multi-frame priors learned during training are naturally leveraged by the recurrent structure during inference.
In scenes with moving subjects against overexposed backgrounds, FreeMEF effectively avoids ghosting, a scenario where parallel fusion methods typically fail.

Highlights & Insights¶

The discovery and resolution of the similarity paradox is the biggest highlight. In domains like HDR where attention metrics and restoration goals are inversely related, blending queries with an extreme region map is an elegant and generalizable concept that can be transferred to other image restoration tasks (e.g., heavily degraded regions in deraining or dehazing).
The recurrent fusion replacing parallel fusion in the "There and Back Again" paradigm: aggregating varying multi-frame information into a unified representation before guided restoration is more natural than step-by-step cross-attention or simple concatenation. This fundamentally accommodates an arbitrary number of frames—representing an architectural paradigm shift rather than a mere engineering trick.
ASE (Attention State Space Equation) is introduced into the recurrent unit as the backbone based on MambaIRv2, validating the effectiveness of hybird SSM and attention in low-level vision recurrent modeling.

Limitations & Future Work¶

The authors acknowledge slight color casting in scenes with extreme differences in exposure levels, where the massive exposure variance drives the color away from the ground truth (as shown in Fig. 11).
The current framework requires all input frames to be aligned to the reference frame coordinate system (via deformable convolutions), where registration may still fail in dynamic scenes with severe motion.
The order of fusion affects quality—requiring the selection of an appropriate input sequence for frames in real-world deployment (optimally placing dark frames last), which adds an extra step of preprocessing logic.

vs SAFNet / AFUNet: These methods are customized for 3-frame inputs, performing concatenation/fusion of all features prior to attention, and thus fail to generalize to other frame counts. FreeMEF fundamentally breaks this limitation via its recurrent architecture.
vs MambaIRv2 / Restormer: These general image restoration methods also support variable-length inputs, but they require adjustments in the fusion layer (along the channel dimension) and subsequent retraining. FreeMEF can perform inference on different frame counts without any architectural modifications.
vs HDR-Transformer / SCTNet: These employ cross-attention for reference-support frame interactions, thereby suffering from the similarity paradox. FreeMEF's EAHA explicitly compensates for this defect using the extreme region map.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Pioneering code-flexible frame-rate MEF architecture; the discovery of the similarity paradox and the proposed EAHA solution are highly ingenious.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 3 datasets, cross-dataset generalization, frame-count generalization, fusion order analysis, and comprehensive ablation studies.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation; the narratives of the "similarity paradox" and "there and back again" are highly compelling.
Value: ⭐⭐⭐⭐⭐ Resolves practical pain points in MEF deployment with a simple and generalizable design.