FreeMEF: A Flexible Multi-Exposure Fusion Transformer for Arbitrary Number of Frames

Conference: ECCV2026
arXiv: 2606.27905
Code: https://github.com/qulishen/FreeMEF
Area: Image Restoration
Keywords: Multi-Exposure Fusion, HDR Imaging, State Space Models, Attention Mechanisms, Variable Frame Count

TL;DR

FreeMEF proposes a "there and back again" two-stage paradigm: first, a recurrent state space module is utilized to progressively aggregate multi-exposure features of arbitrary frame counts into a global representation, which then guides reference frame restoration via extremist-aware hybrid attention. This significantly suppresses ghosting artifacts and enhances dynamic range while supporting flexible inference for 2/3/5 frames.

Background & Motivation

Multi-Exposure Fusion (MEF) reconstructs an HDR image by fusing multiple LDR images of the same scene captured at different exposure values, acting as a classic strategy to overcome the limited dynamic range of sensors. Although deep learning-based MEF methods have made remarkable progress in recent years, they generally suffer from a fundamental limitation: they assume a fixed exposure strategy, meaning the model architectures are specifically designed for a fixed number of frames (e.g., 2, 3, or 5 frames). In practical deployments, different devices or different shooting modes on the same device use varying exposure frame counts. This implies that developers must train and deploy individual models for each frame count, which severely restricts practical efficiency.

A deeper conflict lies in the operation of attention mechanisms. Existing methods—whether fusing first then applying self-attention, or applying cross-attention between a reference frame and auxiliary frames—are essentially based on similarity matching: the query vector originates from the reference frame features, and the key-value vectors come from the auxiliary frame features. However, the core goal of HDR imaging is to recover details in the over- or under-exposed regions of the reference frame, where pixel values are clipped or saturated. These regions exhibit extremely low similarity to the normally exposed regions in the auxiliary frames. This leads to a similarity paradox: the attention mechanism assigns the lowest weights to the regions that need restoration the most. Furthermore, existing parallel fusion strategies tend to blend misalignment information caused by motion between frames, making it difficult to completely delete ghosting in subsequent processing.

The Core Idea of this work is to decompose multi-exposure fusion into two steps: "aggregate first, guide later." First, progressively aggregate information from an arbitrary number of frames recurrently into a global representation, and then use this global representation to guide the restoration of the reference frame in an extremist-aware manner. This "there and back again" paradigm naturally decouples the constraint of input frame counts from the network architecture. It also bypasses the similarity paradox by explicitly learning extremist region maps, allowing the attention mechanism to automatically switch to self-associative search in saturated regions.

Method

Overall Architecture

The overall pipeline of FreeMEF is divided into two main stages. Given a set of exposure frames \(\{\mathbf{I}_t\}_{t=0}^T\) (\(\mathbf{I}_0\) as the reference frame, and the rest as auxiliary frames), the first stage employs a Recurrent State Space Module (RSSM) to process frame-by-frame in arbitrary order. It progressively aggregates global information into a compact global fused feature \(\mathbf{H}_T\) via deformable alignment and state space recurrence. The second stage takes the reference frame \(\mathbf{I}_0\) and the global feature \(\mathbf{H}_T\) as input, feeding them into a U-shaped Transformer encoder-decoder architecture. A Global Feature Guided Block (GFGB) is deployed at each resolution level to selectively leverage global context for restoring the reference frame. Inside the GFGB, two sub-modules are designed: Extremist-Aware Hybrid Attention (EAHA) to address the similarity paradox, and an Affine-Injected Feed-Forward Network (AFFN) for explicit adjustment of brightness and contrast.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Exposure Frames<br/>I0, I1, ..., IT"] --> B["Shared Feature Extraction<br/>FEM"]
    B --> C["RSSM: Frame-by-Frame Recurrent Fusion<br/>Deformable Alignment + ASE<br/>Gated Update via Sigmoid"]
    C --> D["Global Fused Feature HT"]
    D --> E["GFGB × Multi-scale<br/>U-shape Encoder-Decoder"]
    E --> F["EAHA: Extremist-Aware<br/>Hybrid Attention"]
    E --> G["AFFN: Affine Modulation<br/>Brightness/Contrast Adjustment"]
    F --> H["Fused Result I_hat"]
    G --> H

Key Designs

1. RSSM: Recurrent Fusion Mechanism Based on State Space

The RSSM is the core of FreeMEF's capability to support arbitrary frame counts. Each input frame \(\mathbf{I}_t\) first passes through a shared lightweight Feature Extraction Module (FEM) to obtain shallow features \(\mathbf{F}_t\), which then enter the recurrent unit. The recurrent unit performs two key operations: First, it predicts offsets using deformable convolutions (DCN) based on the historical state \(\mathbf{H}_{t-1}\) and current features \(\mathbf{F}_t\), registering the aligned current features \(\bar{\mathbf{F}}_t\) into the coordinate system of the historical state. This handles spatial misalignment caused by handheld camera shake or scene motion between different exposure frames. Second, the aligned features are fed into the Attention State Space Equation (ASE). Since the fixed output matrix \(\mathbf{C}\) in standard SSMs limits the receptive field to the causal scanning direction, ASE dynamically selects semantic prototypes for each pixel via a learnable prompt pool, expanding the output matrix to \(\mathbf{C}+\mathbf{P}\). This introduces non-causal global context while preserving linear complexity. After gated activation, the output of ASE is weight-fused with the historical state through a sigmoid gating mechanism \(\mathbf{G}_t\): \(\mathbf{H}_t = \mathbf{H}_{t-1} + \mathbf{G}_t \odot (\text{update candidate} - \mathbf{H}_{t-1})\). This recurrent architecture naturally fits any number of frames—trained on 3 frames, it can seamlessly switch to 2 or 5 frames during inference without any architectural changes.

2. EAHA: Extremist-Aware Hybrid Attention Resolving the Similarity Paradox

This is the most ingenious contribution of the paper. The reference frame has extreme values and degraded structures in over- or under-exposed regions. When query vectors \(\mathbf{Q}_{base}\) generated directly from the reference frame features are used to match the \(\mathbf{K}\) of global features, the similarity is inherently low, meaning attention weights unfortunately fall on regions that need restoration the most. EAHA resolves this by introducing a second query path \(\mathbf{Q}_{ref}\), which is generated directly from the global fused feature \(\mathbf{H}_T\) as a fallback self-associative search vector within the global representation. Crucially, an extremist region map \(\mathbf{E}=\sigma(\text{DW-Conv}(\mathbf{F}_0))\) is utilized to control the hybrid ratio pixel-by-pixel: \(\mathbf{Q}_{hybrid} = (1-\mathbf{E})\odot\mathbf{Q}_{base} + \mathbf{E}\odot\mathbf{Q}_{ref}\). In normally exposed regions (\(\mathbf{E}\approx 0\)), the model queries using the reference frame content; in saturated regions (\(\mathbf{E}\approx 1\)), it automatically switches to associative search within the global features. This "soft switching" enables the attention mechanism to find meaningful matches in all regions. Additionally, when attention is computed along the channel dimension, \(\mathbf{K}^\top\mathbf{Q}_{hybrid}\) is equivalent to the weighted sum of self-attention and cross-attention, introducing no extra computational company.

3. AFFN: Affine-Injected Feed-Forward Network for Explicit Brightness Regulation

Standard FFNs lack the awareness of global exposure shifts. The intuition of AFFN is that LDR-to-HDR restoration is not a simple addition of features, but requires explicit modification of brightness statistics (scale and shift). It extracts a global descriptor from the global fused feature \(\mathbf{H}_T\)—first applying global average pool, passing through a convolution, and splitting into a scaling parameter \(\gamma\) and a shift parameter \(\beta\). It then applies an affine transformation to the input features: \(\mathbf{X} \odot (1+\gamma) + \beta\), before feeding them into a gated linear unit. This mechanism enables the model to explicitly adjust local feature contrast and color shifts based on global high-dynamic-range information, resulting in more natural brightness and fewer color casts in the visual outputs.

Loss & Training

The network is trained using the standard \(L_1\) loss. The optimizer is Adam (\(\beta_1=0.9,\beta_2=0.999\)), with an initial learning rate of \(2\times 10^{-4}\) decaying under a cosine annealing restart scheduler for a total of 300k iterations.

Key Experimental Results

Main Results

Dataset	Metric	FreeMEF	Prev. SOTA (AFUNet)	Gain
Kalantari et al.	PSNR↑ / SSIM↑ / LPIPS↓	28.418 / 0.948 / 0.081	27.226 / 0.925 / 0.091	+1.19dB PSNR
Real-HDRV	PSNR↑ / SSIM↑ / LPIPS↓	26.077 / 0.940 / 0.074	25.562 / 0.924 / 0.089 (Restormer)	+0.52dB PSNR

Cross-dataset generalization (SICE dataset, trained on Kalantari, directly evaluated on different frame counts):

Test Frames	Metric	FreeMEF	Prev. SOTA	Gain
2 frames	PSNR↑ / SSIM↑	17.087 / 0.731	15.339 / 0.690	+1.75dB
3 frames	PSNR↑ / SSIM↑	19.326 / 0.774	18.724 / 0.768	+0.60dB
5 frames	PSNR↑ / SSIM↑	22.269 / 0.851	21.153 / 0.842	+1.12dB

FreeMEF has 8.9M parameters and 41.5G FLOPs, saving 56% of computation compared to HDR-Transformer.

Ablation Study

Configuration	PSNR	SSIM	Description
Full FreeMEF	28.418	0.948	Full model
RSSM → CNN embedding layer	27.082	0.920	Removing RSSM leads to a 1.34dB drop, demonstrating the value of recurrent fusion
EAHA → MDTA	27.565	0.937	Removing EAHA leads to a 0.85dB drop, showing the importance of extremist-aware hybrid attention
AFFN → GDFN	27.982	0.937	Removing AFFN leads to a 0.44dB drop, highlighting the significant contribution of affine modulation

Key Findings

• RSSM contributes the most (+1.34dB), showing that recurrent progressive fusion is significantly superior to simple parallel processing after CNN embedding.
• Fusion order experiments reveal that placing dark frames (long exposure) at the end of the fusion pipeline helps optimize final image quality, while placing bright frames at the end introduces more noise.
• The method demonstrates unique advantages in cross-dataset generalization: achieving a +1.75dB gain in the 2-frame scenario, because the multi-frame priors learned during training are naturally utilized by the recurrent structure during inference.
• In scenes with moving subjects in overexposed backgrounds, FreeMEF effectively avoids ghosting, a situation where parallel fusion methods commonly fail.

Highlights & Insights

• The discovery and solution of the similarity paradox is the biggest highlight. In fields like HDR where attention measurement runs contrary to the restoration target, mixing queries using extremist maps is an elegant and generalizable idea, which could be extended to other image restoration tasks (such as heavily degraded regions in deraining or dehazing).
• The "there and back again" paradigm of replacing parallel fusion with recurrent fusion: aggregating heterogeneous multi-frame information into a unified representation before guided restoration is more natural than step-by-step cross-attention or simple concatenation, and naturally accommodates arbitrary frame counts—this is an architectural level shift, not just a trick.
• ASE (Attention State Space Equation) is introduced into the recurrent unit as a backbone on top of MambaIRv2, validating the effectiveness of hybrid SSM+attention for recurrent modeling in low-level vision.

Limitations & Future Work

• The authors acknowledge slight color casts in scenarios with extreme exposure differences, as the massive exposure gap causes colors to drift from the real scene (as shown in Fig. 11).
• The current framework requires all input frames to be aligned to the reference frame coordinate system prior (via deformable convolutions); registration may still fail in dynamic scenes with severe motion.
• Fusion order affects quality—in practical deployment, a proper frame input sequence needs to be chosen (optimally placing dark frames last), which adds an extra step of preprocessing logic.

Related Work & Insights

• vs SAFNet / AFUNet: These methods are specifically designed for 3-frame inputs, performing fusion/concatenation of all features before applying attention, making them unable to generalize to other frame counts. FreeMEF fundamentally breaks this restriction with a recurrent architecture.
• vs MambaIRv2 / Restormer: These general image restoration methods also support variable-length inputs, but they require adjustments in the fusion layer (along the channel dimension) and retraining. FreeMEF can perform inference with varying frame counts without any architecture modification.
• vs HDR-Transformer / SCTNet: These methods use cross-attention for reference-to-auxiliary frame interaction, suffering from the similarity paradox. FreeMEF's EAHA explicitly addresses this limitation using extremist region maps.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering variable-frame-count MEF architecture; the discovery of the similarity paradox and the EAHA solution are highly ingenious.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 3 datasets, cross-dataset generalization, frame-count generalization, fusion order analysis, and extensive ablation studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation; the narratives of "similarity paradox" and "there and back again" are highly compelling.
• Value: ⭐⭐⭐⭐⭐ Solves a practical pain point in MEF deployment with a simple, generalizable design.

Related Papers

• [ECCV 2026] Spectral and Trajectory Regularization for Diffusion Transformer Super-Resolution
• [ECCV 2024] Restoring Images in Adverse Weather Conditions via Histogram Transformer
• [ICCV 2025] Devil is in the Uniformity: Exploring Diverse Learners within Transformer for Image Restoration
• [ICCV 2025] Enhancing Image Restoration Transformer via Adaptive Translation Equivariance
• [CVPR 2025] Progressive Focused Transformer for Single Image Super-Resolution