MArFE: Multi-Contrast MRI Arbitrary Scale Super-Resolution with Fourier Enhancement

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/zhiwen-shi/MArFE
Area: Medical Imaging
Keywords: Multi-contrast MRI / Arbitrary-scale super-resolution / Implicit neural representation / Fourier reparameterization / Linear attention neural operator

TL;DR

To alleviate the inherent spectral bias of implicit neural representations and insufficient multi-contrast spatial feature fusion, this paper introduces MArFE, a Fourier-enhanced arbitrary-scale multi-contrast MRI super-resolution framework incorporating fixed Fourier basis reparameterization, a non-local Galerkin linear attention operator, and dual-domain losses for continuous-scale high-frequency detail recovery.

Background & Motivation

Magnetic Resonance Imaging (MRI) serves as an indispensable modality in clinical diagnostic workflows for neurological, cardiovascular, and musculoskeletal disorders, where standard protocols acquire complementary multi-contrast sequences including T1-weighted, T2-weighted, FLAIR, and proton density-weighted (PDW) images. Due to physical tradeoffs among scan acquisition time, signal-to-noise ratio (SNR), and spatial resolution, capturing full volumetric datasets with isotropic high resolution across all sequences remains clinically intractable. While reference-guided multi-contrast super-resolution (SR) leverages a high-resolution auxiliary sequence to guide low-resolution target sequences, conventional deep learning methods predominantly focus on integer upsampling factors (such as \(\times 2\) or \(\times 4\)), failing to accommodate the continuous, variable slice thicknesses and resolutions required across diverse hospital scanners.

Implicit Neural Representation (INR) models offer an appealing foundation for Arbitrary-Scale Super-Resolution (ASSR) by formulating discrete pixel arrays as continuous coordinate-to-signal intensity mappings. Nevertheless, adapting canonical INR pipelines to multi-contrast MRI reveals two severe bottlenecks. First, coordinate-driven multi-layer perceptrons suffer from severe spectral bias, preferentially fitting low-frequency structural components while discarding high-frequency anatomical textures, resulting in over-smoothed edges and compromised clinical utility. Second, MRI signals are intrinsically acquired in complex-valued frequency space (k-space), yet conventional INR frameworks treat inputs as standard spatial images and fuse multi-contrast modalities through simple channel concatenation, omitting continuous feature-coordinate interaction and failing to exploit the true frequency-domain characteristics of MR physics.

Addressing this disconnect between frequency-domain physics and continuous spatial coordinate decoding, this paper approaches the multi-contrast ASSR problem through complex-valued frequency modeling and neural operator theory. Core idea: formulate multi-contrast arbitrary-scale MRI super-resolution as a complex-valued dual-branch implicit architecture, combining fixed Fourier basis reparameterization and a non-local Galerkin-type linear attention neural operator with dual-domain supervision to resolve spectral bias and accurately reconstruct fine anatomical details across continuous scales.

Method

Overall Architecture

MArFE ingests paired arbitrary-resolution complex-valued MR inputs: a target low-resolution image (Target LR \(\in \mathbb{R}^{H_{tar} \times W_{tar}}\)) and an auxiliary reference image (Reference LR/HR \(\in \mathbb{R}^{H_{ref} \times W_{ref}}\)), treating real and imaginary components as dual channels. A weight-shared Residual Dense Network (RDN) extracts deep multi-level features from both branches, which are subsequently interpolated to unified spatial dimensions based on their respective scale factors. The aligned feature maps and continuous relative positional encodings are injected into the implicit neural decoder, passing through the Fourier Reparameterization Module (FRM), the Non-Local Galerkin-type Linear Attention (NLGLA) operator, and recurrent contrast-aware residual feedback blocks to output the target high-resolution reconstruction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Complex MRI Inputs<br/>Target LR & Ref LR/HR"] --> B["Feature Extraction & Alignment<br/>Weight-shared RDN + Scale-factor interpolation"]
    B --> C["Coordinate-Feature Fusion<br/>Relative positional encoding concatenation"]
    C --> D["Fourier Reparameterization<br/>Fixed orthogonal basis decomposition + Sin activation"]
    D --> E["Non-Local Galerkin Linear Attention<br/>Kernelized FNO with O(N) complexity"]
    E --> F["Contrast-Aware Dense Implicit Decoder<br/>Decoder+ feature loop-back & residual blocks"]
    F --> G["Super-Resolved Outputs & Dual-Domain Loss<br/>Target/Ref SR with joint GKLoss / HFLoss"]

Key Designs

1. Fourier Reparameterization: Fixed Orthogonal Basis Decomposition with Sinusoidal Activation

Standard coordinate networks utilizing ReLU activations struggle to retain high-frequency sensitivity during back-propagation. Exploiting the property that MR images exhibit structured, concentrated frequency distributions, the Fourier Reparameterization Module (FRM) decomposes features into fixed 2D Fourier orthogonal sine and cosine basis matrices \(B\). The projection weight matrix is factorized as \(W = \Lambda B^T\), where \(\Lambda\) is a trainable coefficient matrix. Followed by a sinusoidal activation function in a SIREN-like configuration, this explicit frequency-domain reparameterization enhances the network's responsiveness to high-frequency medical textures and prevents spectral bias degradation during deep coordinate mapping.

2. Non-Local Galerkin Linear Attention: Operator Kernel Integration with \(O(N)\) Complexity

While Fourier Neural Operators (FNO) effectively learn mappings between continuous infinite-dimensional function spaces across arbitrary resolutions, vanilla FNO integration incurs heavy computational burdens during dense INR decoding. The proposed Non-Local Galerkin-type Linear Attention (NLGLA) reformulates the continuous FNO integral kernel into a linear attention mechanism via Galerkin projection. By introducing a positive kernel feature mapping \(\phi(\cdot)\) onto query \(Q\), key \(K\), and value \(V\) matrices: $\(\mathrm{Attention}(Q, K, V) = \phi(Q) \left(\phi(K)^T V\right)\)$ the computational complexity drops from quadratic \(O(N^2)\) to linear \(O(N)\) with respect to spatial token count \(N\). This formulation enables long-range spatial context modeling and global spectral interactions, maintaining structural consistency and fine high-frequency fidelity without exceeding memory constraints.

3. Contrast-Aware Dense Implicit Decoder: Loop-Back Coordinate Recurrence and Residual Enhancement

Conventional implicit decoders inject multi-contrast reference representations solely through an initial global concatenation, causing deeper decoding layers to progressively lose coordinate-level geometric constraints. The improved dense decoder (Decoder+) incorporates a feature loop-back mechanism and residual enhancement units. Relative positional encodings and multi-scale RDN features are recurrently fed into intermediate decoding stages, reinforcing the non-linear calibration between spatial coordinate grids and pixel intensity flows, ensuring reference anatomical priors are thoroughly transferred into target sequence textures.

Loss & Training

To exploit the complementary constraints of spatial image space and k-space frequency data, MArFE is trained under a joint dual-domain objective: 1. Global K-space Loss (GKLoss): Computes a weighted \(L_1\) penalty in the Fourier domain. Given frequency coordinates \((i, j)\) and distance to the DC center \(d(i, j) = \sqrt{(i - h/2)^2 + (j - w/2)^2}\), a distance-aware weight \(w(i, j) = e^{\alpha \cdot \frac{d(i, j)}{\max d(i, j)}}\) penalizes peripheral high-frequency errors, forcing the network to restore truncated k-space components. 2. Spatial High-Frequency Loss (HFLoss): Uses a Gaussian low-pass filter (\(\sigma = 0.5\)) to isolate low-frequency content. The high-frequency residual is obtained by subtracting the low-frequency component from the spatial image, and penalized via an \(L_1\) loss to enforce crisp anatomical boundaries. 3. Total Loss Function: $\(\mathcal{L} = \mathcal{L}_{L1} + \lambda_1 \mathcal{L}_{\text{gkloss}} + \lambda_2 \mathcal{L}_{\text{hfloss}}\)$ where \(\lambda_1 = 0.05\) and \(\lambda_2 = 0.05\). Optimization uses Adam with an initial learning rate of \(10^{-4}\) (halved every 40 epochs) over 200 epochs on a single GPU, training on \(32 \times 32\) patches with random downsampling scale factors uniformly sampled from \([1, 4]\) at step 0.1.

Key Experimental Results

Main Results

Quantitative evaluations across in-distribution (\(\times 1.5, \times 2, \times 3, \times 4\)) and out-of-distribution (\(\times 6, \times 8\)) scale factors on the fastMRI dataset (PDW reference, FS-PDW target) and IXI dataset (PDW reference, T2W target), using the standard RDN feature extractor for all ASSR baselines:

Dataset	Method / Model	\(\times 1.5\) (PSNR/SSIM)	\(\times 2\) (PSNR/SSIM)	\(\times 4\) (PSNR/SSIM)	\(\times 8\) (OOD, PSNR/SSIM)	Average (PSNR/SSIM)
fastMRI	Bicubic	34.799 / 0.912	30.719 / 0.822	26.415 / 0.514	23.337 / 0.432	15.527 / 0.737
fastMRI	McMRSR (Fixed scale)	37.773 / 0.947	34.546 / 0.884	30.141 / 0.734	26.200 / 0.547	17.417 / 0.842
fastMRI	WavTrans (Fixed scale)	36.390 / 0.934	32.841 / 0.859	30.197 / 0.741	26.722 / 0.585	17.242 / 0.919
fastMRI	Meta-SR (ASSR)	39.974 / 0.948	34.354 / 0.883	30.085 / 0.740	24.483 / 0.498	17.499 / 0.881
fastMRI	LIIF (ASSR)	37.314 / 0.939	34.170 / 0.877	30.232 / 0.736	25.318 / 0.512	17.331 / 0.889
fastMRI	SRNO (ASSR)	38.348 / 0.951	34.728 / 0.888	30.348 / 0.744	25.023 / 0.625	17.482 / 0.897
fastMRI	DualArb (ASSR)	38.166 / 0.948	34.686 / 0.884	30.578 / 0.743	26.250 / 0.549	17.607 / 0.963
fastMRI	MArFE (Ours)	38.449 / 0.951	34.968 / 0.887	31.080 / 0.757	26.871 / 0.598	17.879 / 0.964
IXI	Bicubic	33.165 / 0.967	28.267 / 0.906	22.958 / 0.703	20.650 / 0.486	14.088 / 0.650
IXI	WavTrans (Fixed scale)	39.118 / 0.966	38.171 / 0.929	35.804 / 0.940	27.832 / 0.825	19.484 / 0.762
IXI	LIIF (ASSR)	41.886 / 0.992	35.557 / 0.973	29.458 / 0.904	24.009 / 0.712	17.634 / 0.750
IXI	HIIF (ASSR)	43.413 / 0.993	38.020 / 0.980	27.650 / 0.891	23.478 / 0.720	17.812 / 0.760
IXI	DualArb (ASSR)	44.691 / 0.994	41.187 / 0.986	36.863 / 0.969	28.681 / 0.906	20.702 / 0.763
IXI	MArFE (Ours)	46.663 / 0.996	42.317 / 0.988	38.041 / 0.973	30.625 / 0.905	21.492 / 0.779

In downstream fully automatic brain anatomical segmentation (ANTs Atropos; CSF, GM, and WM), MArFE outputs achieve the highest fidelity to ground truth segmentations, attaining average Dice scores of 0.957 at \(\times 2\) (surpassing DualArb's 0.950 and WavTrans's 0.946) and 0.928 at \(\times 4\) (surpassing DualArb's 0.910), verifying its clinical utility in downstream morphological quantification.

Ablation Study

1. Architectural Module Ablation (fastMRI and IXI)

Config Index	Decoder+	NLGLA	FRMa (post-ReLU)	FRMb (post-Sin)	fastMRI Avg PSNR (dB)	fastMRI Avg SSIM	IXI Avg PSNR (dB)	IXI Avg SSIM
1 (Baseline)	\(\times\)	\(\times\)	\(\times\)	\(\times\)	31.608	0.763	37.142	0.963
2	\(\checkmark\)	\(\times\)	\(\times\)	\(\times\)	31.861	0.770	37.986	0.964
3	\(\checkmark\)	\(\checkmark\)	\(\times\)	\(\times\)	32.054	0.774	38.343	0.969
4	\(\checkmark\)	\(\times\)	\(\times\)	\(\checkmark\)	31.973	0.766	38.319	0.968
5	\(\checkmark\)	\(\checkmark\)	\(\checkmark\)	\(\times\)	31.896	0.769	38.576	0.968
6 (Full Model)	\(\checkmark\)	\(\checkmark\)	\(\times\)	\(\checkmark\)	32.099	0.779	38.588	0.970

2. Loss Function Ablation

Loss Setup	fastMRI \(\times 4\) PSNR	fastMRI \(\times 8\) PSNR	fastMRI Avg PSNR	fastMRI Avg SSIM	IXI \(\times 4\) PSNR	IXI \(\times 8\) PSNR	IXI Avg PSNR	IXI Avg SSIM
\(L_1\) Baseline	30.998	24.057	31.029	0.744	38.037	30.441	38.356	0.966
\(L_1 + \text{HFLoss}\)	30.962	26.011	31.803	0.764	37.928	30.462	38.411	0.967
\(L_1 + \text{GKLoss}\)	30.963	26.150	31.868	0.767	38.033	30.133	38.473	0.968
Full Combined Loss	31.080	26.871	32.099	0.779	38.041	30.625	38.588	0.970

Key Findings

• Impact of Fourier Basis Placement and Non-Linear Activation: Comparing FRMa (Fourier basis applied after ReLU) with FRMb (applied after sinusoidal activation), FRMb consistently attains superior metrics (reaching 32.099 dB on fastMRI). This confirms that pairing orthogonal frequency projection with sinusoidal activation provides effective inductive bias against spectral bias in continuous coordinate fields.
• Robustness in Out-of-Distribution (OOD) Extrapolation: Under extreme \(\times 8\) upsampling where traditional INR baselines collapse (e.g., LIIF drops to 24.009 dB on IXI), MArFE maintains 30.625 dB / 0.905 SSIM. The k-space distance-weighted GKLoss provides the primary stabilizing constraint, boosting \(\times 8\) PSNR by over 2.8 dB compared to pure \(L_1\) supervision.

Highlights & Insights

• Operator-Theoretic Linear Attention: Transforming continuous Fourier Neural Operator kernel integration into an \(O(N)\) linear attention mechanism bridges continuous function space modeling and scalable transformer decoding for medical imaging.
• Physics-Informed Dual-Domain Supervision: Bypassing ad-hoc image spatial heuristics, GKLoss directly penalizes peripheral k-space energy discrepancies based on physical distance to the k-space center, delivering substantial high-frequency gains at negligible hyper-parameter cost (\(\lambda=0.05\)).
• Generalizable Coordinate Paradigm: The decoupled design of FRM and NLGLA can be readily extended to sparse-view CT reconstruction, continuous PET resolution recovery, and arbitrary-scale physical field simulations.

Limitations & Future Work

• Computational Overhead on 3D Volumes: While NLGLA exhibits linear complexity, recurrent frequency conversions and dual-channel complex representations incur higher memory consumption than standard 2D convolutions when rendering large volumetric MRI scans.
• Rigid Registration Assumption: The framework assumes pre-aligned multi-contrast sequences. Subject motion, soft-tissue deformations, or inter-scan phase shifts in clinical environments may cause feature mismatch artifacts.
• Future Directions: The author identifies multi-modal structural-functional super-resolution (e.g., fMRI and DTI) and integration of motion-robust registration mechanisms as fruitful avenues for translation to practical radiology workflows.

Related Work & Insights

• vs LIIF [CVPR 2021] / Meta-SR [CVPR 2019]: Generic ASSR approaches rely on local coordinate regressions that succumb to spectral bias in MR textures; MArFE integrates orthogonal Fourier basis decomposition and dual-domain losses to resolve high-frequency anatomical attenuation.
• vs SRNO [CVPR 2023] / HIIF [CVPR 2025]: While both explore neural operators or frequency representations, they remain confined to single-image super-resolution; MArFE specifically engineers a dual-branch recurrent decoder and Galerkin linear attention to harvest multi-contrast cross-sequence correlations.
• vs Dual-Arb [MICCAI 2023]: Dual-Arb pioneered multi-contrast ASSR using basic feature concatenation and positional encodings; MArFE establishes frequency-operator foundations and k-space supervision, delivering decisive improvements across both in-distribution and large OOD scales.

Rating

• Novelty: ⭐⭐⭐⭐☆ Incorporates Galerkin-type FNO linear attention and Fourier basis reparameterization into continuous multi-contrast MRI super-resolution.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous benchmarks on fastMRI and IXI datasets across six upsampling scales, paired with downstream brain tissue segmentation validation and comprehensive module/loss ablations.
• Writing Quality: ⭐⭐⭐⭐☆ Well-structured exposition linking MRI physical acquisition principles with continuous neural operator mathematical formulations.
• Value: ⭐⭐⭐⭐⭐ Addresses the critical clinical challenge of long MRI acquisition times and thick-slice sequence degradation, accompanied by fully open-source code.