UHD-MFF: Shattering Barriers in Multi-Focus Ultra-High-Definition Image Fusion via Learnable Lookup Tables¶

Conference: ECCV2026
arXiv: 2606.31242
Code: https://github.com/zyb5/UHD-MFF
Area: Image Restoration
Keywords: Multi-Focus Image Fusion, Ultra-High-Definition (UHD), Learnable Lookup Tables, Scale Decoupling, Mobile Deployment

TL;DR¶

This paper proposes the first 4K ultra-high-definition multi-focus image fusion dataset, UHD-MFF (1,950 pairs of 3840×2160 images), and designs a scale-decoupled learnable lookup table framework, UMF-LUT. It uses a low-resolution C-LUT for coarse region-level decision-making and a high-resolution D-LUT for edge-level refinement. With only 0.008M parameters, it achieves state-of-the-art fusion quality while enabling real-time 4K inference at 90fps, allowing deployment on mobile devices.

Background & Motivation¶

Multi-Focus Image Fusion (MFIF) aims to combine multiple images taken at different focus depths in the same scene into an all-in-focus image, which is widely applied in mobile photography, microscopy, and other fields. Over the past decade, MFIF techniques have made significant progress on low-resolution datasets (typically below 1080P), evolving from traditional methods (DSIFT, GFDF) to deep learning models (MFF-GAN, MSFIN-Fusion) and unified frameworks (U2Fusion, TC-MoA). However, as imaging devices enter the 4K/8K era, existing methods face three major barriers in UHD scenarios.

First is the data availability barrier: mainstream datasets like Lytro (520×520), Real-MFF (625×433), and MFI-WHU (600×400) have extremely low resolutions, lacking large-scale 4K training/testing benchmarks. Models trained on low-resolution data degrade severely when generalizing to UHD scenarios. Second is the model adaptability barrier: existing methods directly apply standard convolutions or attention mechanisms to process 4K images—the memory overhead of full-resolution feature maps far exceeds GPU limits, while simple downsampling loses the fine-grained structures at focus boundaries. Finally, there is the deployment feasibility barrier: although the demand for MFIF on mobile devices is immense, current methods mostly rely on computationally intensive servers (e.g., TC-MoA takes 5.7 seconds per image with 340M parameters) and cannot run on phones, which severely limits practical applications.

The Core Idea of this paper is to decompose UHD fusion into low-resolution region-level coarse decision-making and high-resolution edge-level refinement using a scale-decoupled LUT strategy. By replacing full-resolution convolutions with learnable lookup tables, this approach simultaneously shatters the data, adaptability, and deployment barriers.

Method¶

Overall Architecture¶

The Mechanism of UMF-LUT is as follows: instead of performing heavy computations at full resolution, it makes robust region-level focus decisions in a low-resolution space and then compensates for edge details at high resolution using a lightweight LUT. The framework consists of two branches: C-LUT (Coarse-Region Lookup Table) and D-LUT (Detail-Edge Lookup Table), whose outputs are fused and then weighted to synthesize the final image.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Near-focus / Far-focus 4K image pairs"] --> B["Downsampling ×s<br/>Enter low-resolution space"]
    B --> C["C-LUT: Coarse-Region Lookup Table"]
    C --> D["4 Query Cues<br/>Gradient Difference / Dilated Gradient / Max Gradient / Learnable Semantics"]
    D --> E["4D LUT Query +<br/>Quadrilinear Interpolation"]
    E --> F["Upsampling back to<br/>Full-Resolution"]
    F --> G["Weighted Synthesis (Near × M + Far × (1-M))"]
    G --> H["Output: All-in-focus fused image"]

    A --> I["D-LUT: Detail-Edge Lookup Table"]
    I --> J["2 Query Cues<br/>Laplacian Difference / Max Laplacian"]
    J --> K["2D LUT Query +<br/>Bilinear Interpolation"]
    K --> L["Edge Residual ΔM"]
    L --> F

Key Designs¶

1. C-LUT: A 4D coarse-region lookup table for low-resolution focus decision-making

C-LUT operates in a low-resolution space (downsampled by factor \(s\) from the 4K input), converting the decision of "which region is in focus" into a joint query of four complementary cues.

Each of the four cues has its own role: the gradient difference cue (difference between near- and far-focus Sobel gradient maps, reflecting local sharpness differences) and the dilated gradient cue (difference of gradient maps averaged over a local window to suppress noise) focus on the direct sharpness contrast between the two images; the maximum gradient cue (pixel-wise maximum of individual gradient maps to characterize texture richness) helps \(disambiguate\) in low-texture regions; and the learnable regional semantic cue (using a lightweight CNN to extract contextual features from concatenated low-resolution images) complements the gradient cues by capturing scene-level semantics. Each of the four cues is normalized to a discrete index range [0, S_C-1] to form a 4D coordinate (c1, c2, c3, c4) to query a pre-initialized, end-to-end trainable 4D LUT.

During query, since the coordinates are continuous values while the LUT grid points are discrete, quadrilinear interpolation is used—splitting each coordinate into integer indices and fraction residuals, and taking the weighted average of the surrounding \(2^4 = 16\) grid points as output. This design ensures that the entire pipeline is differentiable for training and ensures smooth weight transitions.

2. D-LUT: A 2D detail-edge lookup table for high-resolution boundary refinement

Although C-LUT can make correct region-level decisions at low resolution, the downsampling process naturally loses the precise location of focus boundaries. As a complementary branch, D-LUT runs directly at the native full resolution, focusing on high-frequency residual compensation at boundaries.

Unlike C-LUT which uses a first-order Sobel operator, D-LUT employs a second-order Laplacian operator. Second-order responses emphasize sharp intensity changes and zero-crossing structures, which are naturally suited for boundary localization, whereas first-order gradients are prone to over-smoothing at boundaries. After applying Laplacian kernels to the full-resolution near/far-focus images, two cues are extracted: the Laplacian difference cue (the difference in Laplacian responses between the two images to capture high-frequency focus transitions) and the maximum Laplacian cue (the pixel-wise maximum of Laplacian responses to assess local structural strength and suppress unstable responses in flat regions).

These two cues are normalized into 2D coordinates to query a 2D LUT (using bilinear interpolation with 4 surrounding grid points) to obtain the edge residual ΔM. Through sparsity regularization (\(L1\) norm constraint), D-LUT is trained to activate only at genuine focus boundaries, maintaining near-zero outputs in flat regions—making the computational overhead of D-LUT negligible and its contribution highly focused.

3. Cross-scale decision fusion and unsupervised training strategy

After C-LUT outputs the coarse decision map M_main, it is bilinearly upsampled to full resolution and added pixel-wise to the edge residual ΔM from D-LUT (clamped to the [0,1] range) to obtain the final fusion decision map M_final. The final image is synthesized as M_final · I_near + (1 - M_final) · I_far—a smooth fusion where in-focus areas favor the near-focused image, and out-of-focus areas favor the far-focused image.

For training, C-LUT and D-LUT are optimized jointly in a unified framework. The loss function consists of three parts: the reconstruction loss (weighted sum of intensity \(L1\), gradient \(L1\), and SSIM) measures the unsupervised fidelity of the fused image to the clear reference; the binarization loss E[M_coarse · (1 - M_coarse)] forces C-LUT to make decisive 0/1 choices in flat regions to suppress ghosting artifacts; and the sparsity loss ||ΔM||_1 ensures D-LUT activates only at focus boundaries and does not fabricate structures in flat areas.

Interestingly, despite having ground-truth all-in-focus images from synthetic data, the authors deliberately adopt an unsupervised training strategy—not directly supervising with clear reference images. Experimental comparison shows that the unsupervised strategy yields quantitative metrics on par with the supervised version, but produces more natural brightness and smoother transitions, avoiding occasional contrast artifacts present in supervised versions.

A Complete Example¶

Take the fusion of two 3840×2160 images, "near-focused bottle" and "far-focused background", as an example. The input pair is first downsampled by 8× to 480×270 and fed into the C-LUT. In this low-resolution space, the four cues encode gradient differences, regional energy differences, texture intensity, and semantic context between the near/far focus. The 4D LUT then outputs a 480×270 coarse decision map (foreground near-focus value close to 1, background far-focus value close to 0). Meanwhile, D-LUT operates at full resolution (3840×2160) using Laplacian operators to extract focus transition signals at the edges, outputting a residual ΔM (about 0.05-0.15) that is non-zero only near the boundary between the bottle cap and the background. Upsampling the coarse decision map back to 3840×2160 and adding ΔM transforms the sharp step transition at the boundary into a smooth gradient. During final synthesis, the bottle body contains >95% of the near-focused image, the background contains <5%, and the 40-60% smooth transition in the boundary region eliminates harsh edges.

Loss & Training¶

C-LUT and D-LUT are jointly optimized using the AdamW optimizer with a batch size of 8 and an initial learning rate of 1e-5. The loss function is defined as:

\[ \mathcal{L}_{\text{C-LUT}} = \mathcal{L}_{rec} + \lambda_1 \mathcal{L}_{bin}, \quad \mathcal{L}_{\text{D-LUT}} = \mathcal{L}_{rec} + \lambda_2 \mathcal{L}_{sparse} \]

where the reconstruction loss is \(\mathcal{L}_{rec} = \lambda_3 \mathcal{L}_{int} + \lambda_4 \mathcal{L}_{grad} + \lambda_5 \mathcal{L}_{ssim}\), and the hyperparameters are set to \(\lambda_1=50,\ \lambda_2=0.05,\ \lambda_3=20,\ \lambda_4=0.7,\ \lambda_5=0.8\).

Key Experimental Results¶

Main Results¶

Dataset	Metric	Ours	Prev. Best	Description
UHD-MFF-Real (150 pairs of 4K real)	MI ↑	4.8029	4.7667 (MSFIN)	Best mutual information
UHD-MFF-Real	VIF ↑	1.3814	1.3628 (SESF)	Best visual information fidelity
UHD-MFF-Real	Q_AB/F ↑	0.7290	0.7145 (MSFIN)	Best gradient-based fusion quality
UHD-MFF-Syn (300 pairs)	SD ↑	49.9062	49.8757 (CCSR-Net)	Leading standard deviation
UHD-MFF-Syn	VIF ↑	1.4335	1.4186 (CCSR-Net)	Leading visual information fidelity

Efficiency Metric	Ours	Prev. Best (SESF)	Description
Inference Time (ms)	11.03	150.35 (SESF)	Real-time 90fps
Parameters (M)	0.0081	0.014 (SESF)	Smallest model
Peak Memory (MB)	1133.15	2109.36 (SESF)	Lowest GPU memory
Energy Consumption (mJ)	67.45	84.07 (SESF)	Lowest power consumption
Mobile Inference Time (s)	0.97	55.68 (SESF, only comparable)	Huawei P60 Pro

Ablation Study¶

Configuration	MI ↑	VIF ↑	Q_AB/F ↑	Description
Full model	6.3999	1.4335	0.7498	Full model
w/o C_GD (gradient diff cue)	6.1312	1.4028	0.7449	Loss of near/far focus contrast signal
w/o C_DG (dilated gradient cue)	5.9196	1.2564	0.6601	Largest drop—regional energy crucial for noise suppression
w/o C_MG (max gradient cue)	6.0500	1.3948	0.7436	Prone to confusion in low-texture regions
w/o C_RS (semantic cue)	5.9962	1.3833	0.7417	Lacks scene-level context
w/o D-LUT (edge refinement)	6.3538	1.4210	0.7501	Edge clarity drops, but quantitative impact is limited

Key Findings¶

Dilated gradient cue contributes the most: Removing it causes the most severe drop across all metrics (VIF drops from 1.4335 to 1.2564), indicating that aggregating neighborhood information to establish spatial consistency in noisy environments is far more robust than simple pixel-wise gradient differences.
Training data scale is crucial: Expanding the training set from 15 pairs to 1500 pairs increases AG from 1.479 to 1.942 (+31%), and Q_AB/F from 0.5367 to 0.7498 (+40%)—indicating that fusion in UHD scenarios indeed requires large-scale high-fidelity data support.
High-resolution training is irreplaceable: Directly inferring on 4K images after training on low-resolution data leads to severe degradation across all metrics (VIF drops from 1.4335 to 1.2237), validating the necessity of the UHD-MFF dataset.
Unsupervised vs. Supervised: Quantitative metrics are close, but the unsupervised version generates more natural brightness and smoother transitions, avoiding occasional contrast distortions and spectral shifts found in the supervised version.
A quantum leap in efficiency: The measured 4K speed of 11ms per image (90fps) in our model is 13.6 times faster than the runner-up SESF, and 518 times faster than the slow TC-MoA—this efficiency difference directly determines whether it can be deployed on mobile devices.

Highlights & Insights¶

Using LUTs to replace full-resolution convolutions reduces the computational cost of UHD fusion from \(O(HW \cdot K^2)\) to \(O(1)\): C-LUT infers in an 8× downsampled space, while D-LUT only queries a 2D table for boundary tuning—this design choice compresses the parameters for 4K fusion to just 8,100 (4-5 orders of magnitude smaller than most neural networks), which is the fundamental reason the model can run in real time on mobile phones.
The concept of scale decoupling is highly generalizable: Decomposing a "high-resolution task" into a dual-scale LUT architecture of "low-resolution coarse decision + high-resolution boundary refinement" can be generalized to other UHD image processing tasks (such as denoising, super-resolution, and HDR fusion), which is particularly friendly to resource-constrained devices.
A clean division of labor between binarization loss and sparsity loss: The former forces C-LUT to make decisive binary choices in flat regions to eliminate ghosting, while the latter keeps D-LUT focused on boundaries instead of fabricating texture in flat areas. These two regularization terms guard the quality of the decision map from different dimensions, with clear design intent and interpretable effects.
Intriguing finding regarding the unsupervised training strategy: Despite the availability of synthetic ground truth, the authors deliberately avoid using it. Empirical results demonstrate that the unsupervised version generates more natural visual qualities while maintaining comparable quantitative metrics, offering fresh inspiration for the training paradigm of image fusion tasks.

Limitations & Future Work¶

The downsampling ratio \(s\) in C-LUT is hardcoded: The paper lacks a detailed ablation study on the choice of \(s\). Different resolutions and scenes may require adaptive ratios. Future work could explore learnable adaptive downsampling strategies.
The dataset is dominated by static scenes: UHD-MFF currently only contains static image pairs, lacking multi-focus fusion data in dynamic scenarios (e.g., moving objects, illumination changes). Video multi-focus fusion is a natural extension.
The Laplacian operator degrades in flat regions: When large textureless regions (e.g., white walls, sky) exist in the image, both Laplacian cues for D-LUT approach zero, rendering its edge refinement capability almost ineffective in these areas. However, since C-LUT's coarse decision is already sufficient in such scenarios, the impact on the final result is limited.
Joint optimization with other image enhancement tasks (super-resolution, HDR) remains unexplored: In real-world mobile camera pipelines, multi-focus fusion is often executed sequentially with super-resolution and HDR. Whether UMF-LUT can be jointly tuned with other modules merits further investigation.

vs TC-MoA (CVPR 2024): TC-MoA utilizes a task-customized MoE adapter for general image fusion. While achieving good performance, it requires 340M parameters and 5.7 seconds to process a 4K image. In contrast, our model, specially designed for UHD scenarios using LUTs, delivers comparable or superior fusion quality with only 8.1K parameters, representing a qualitative leap in efficiency.
vs CCSR-Net: CCSR-Net unrolls coupled convolutional sparse representation into a learnable network, offering search interpretability, but still takes nearly 3 seconds per 4K image. Our LUT strategy performs similarly (and sometimes better) in quantitative metrics while being two orders of magnitude faster.
vs SESF: SESF is the only prior method capable of running on mobile devices, but taking 55.68 seconds to process a 4K image. The proposed method compresses this time to under 1 second, making real-time mobile multi-focus fusion experiences possible.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The first 4K multi-focus fusion dataset coupled with a dual-scale decoupled LUT architecture—offering an innovative approach and clear engineering deployment value.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive ablations conducted across multiple facets (individual cue contributions, data scales, training strategies, and resolution comparisons), with particularly detailed efficiency benchmarks.
Writing Quality: ⭐⭐⭐⭐⭐ Clear narrative structure centered around the three barriers, complete methodological explanation, and professional presentation of experimental comparisons.
Value: ⭐⭐⭐⭐⭐ Beyond advancing state-of-the-art academic benchmarks, it provides a practical engineering solution ready for mobile deployment, holding direct value for the industry.