Fidelity- and Perception-Aware Local Implicit Attention for Arbitrary-Scale Image Super-Resolution¶

Conference: ECCV2026
arXiv: 2606.21910
Code: https://github.com/XUSean0118/FPLIA
Area: Image Restoration
Keywords: Arbitrary-scale image super-resolution, diffusion models, local implicit functions, fidelity-perception trade-off, adaptive feature selection

TL;DR¶

FPLIA proposes a dual-stream framework that fuses the fidelity features of a regression backbone and the perceptual features of a diffusion model via asymmetric bidirectional cross-attention (FPAM) and pixel-wise adaptive selection (FPSM), simultaneously achieving high fidelity and high perceptual quality in ASISR.

Background & Motivation¶

Arbitrary-scale image super-resolution (ASISR) aims to reconstruct high-resolution images across a continuous range of scaling factors using a single model. The current mainstream paradigm combines deep feature extractors with local implicit image functions, mapping 2D query positions and their neighborhood features to RGB pixel values via coordinate-conditioned MLPs. This paradigm faces a fundamental tension: the source of the features determines the upper bound of the final quality. Regression-backbone methods (SwinIR, EDSR, etc.) trained with L1/L2 losses (such as LIIF, LTE, CiaoSR, HIIF) obtain extremely high pixel-level fidelity (high PSNR), but the reconstruction results naturally suppress high-frequency textures, appearing visually blurry. Conversely, diffusion-based methods (such as IDM, Kim et al.) generate perceptually sharp and realistic textures, yet carry the risk of structural hallucinations—potentially generating non-existent characters in repetitive texture regions. These two classes of methods perform distinctly on different evaluation metrics for the exact same image, making it difficult to achieve the best of both worlds.

This conflict between fidelity and perceptual quality cannot be resolved by a simple "middle ground" compromise. The key lies in the fact that the discrepancy in information between the two feature streams is asymmetric: fidelity features provide a structurally anchored scaffolding that constrains the perceptual features toward the true structure, while perceptual features inject high-frequency diversity, supplementing details to the smooth fidelity representation. This suggests that their interaction should be modeled as separate asymmetric pathways rather than simple concatenation or weighted averaging. Meanwhile, the relative importance of these two types of features dynamically changes with spatial locations and scaling factors—fidelity features are sufficient in flat regions, whereas texture-rich regions urgently need perceptual cues; additionally, higher magnification factors reduce the available information in fidelity features, increasing the dependence on perceptual features. Therefore, a good fusion framework requires two capabilities: generating diverse candidate representations covering different fidelity-perception ratios through interaction, and adaptively choosing the most reliable candidate at each query location.

The core idea of this paper is to propose the FPLIA framework. It generates four candidate features with different fidelity-perception ratios at each query coordinate through the Asymmetric Bidirectional Cross-Feature Attention Module (FPAM). Then, it selects the most suitable candidate to predict RGB values based on local content and scale factors via the Pixel-Wise Adaptive Selection Module (FPSM), thereby simultaneously achieving high fidelity and high perceptual quality.

Method¶

Overall Architecture¶

The overall pipeline of FPLIA is divided into five components: a fidelity feature extractor (SwinIR) extracts the fidelity feature map \(\mathcal{F}_f\) from the low-resolution input; a perceptual feature generator (Stable Diffusion v1.5) conditioned on \(\mathcal{F}_f\) produces the perceptual feature map \(\mathcal{F}_p\); both feature maps are input to FPAM, which applies self-attention and bidirectional cross-attention at each query coordinate to produce four candidate representations; FPSM estimates the confidence for each candidate and selects the one with the highest confidence; finally, the decoder MLP maps the selected feature to residual RGB values, which are added to the bilinearly upsampled low-resolution image to produce the final output. The fidelity extractor and perceptual generator are frozen during the fine-tuning phase, and only FPAM, FPSM, and the decoder are updated. Consequently, the computational overhead is minimally increased (less than 5% latency and 2% GPU memory increase compared to the Kim et al. baseline).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Low-Resolution Input I_LR"] --> B1["Fidelity Extractor SwinIR<br/>Outputs fidelity feature F_f"]
    A --> B2["Perceptual Generator Stable Diffusion<br/>Conditioned on F_f, outputs perceptual feature F_p"]
    B1 --> C["FPAM: Dual-Stream Bidirectional Cross-Attention<br/>Generates four candidate features"]
    B2 --> C
    C --> D["FPSM: Pixel-Wise Adaptive Selection<br/>Based on local content and scale factor"]
    D --> E["Decoder MLP → Residual RGB<br/>+ Bilinear Upsampling → HR Output"]

Key Designs¶

1. FPAM: Asymmetric Bidirectional Cross-Feature Attention Generates Diverse Candidates

The core insight of FPAM is that the interaction direction between fidelity and perceptual features is asymmetric, requiring independent pathways rather than a single symmetric operation. It extends the single-stream cross-scale local attention (CSLA) into a dual-stream architecture: for each query coordinate \(x_q\) on the low-resolution feature map, query embeddings \(q_f, q_p\) are interpolated from the two feature maps respectively, and key-value embeddings are sampled from the local neighborhood grid. The standard CSLA function is executed independently under four combinations: (i) fidelity self-attention \(\tilde{f}_f\), (ii) perceptual self-attention \(\tilde{f}_p\), (iii) perceptual query \(\times\) fidelity key-value \(\tilde{f}_{pf}\), and (iv) fidelity query \(\times\) perceptual key-value \(\tilde{f}_{fp}\). Here, \(\tilde{f}_{fp}\) supplements perceptual details on a fidelity-anchored basis, and \(\tilde{f}_{pf}\) constrains structural fidelity in a perception-driven representation. These two occupy different regions of the fidelity-perception spectrum, forming mutually irreplaceable candidate pools. Ablation studies show that using only the two self-attention outputs (\(\tilde{f}_f + \tilde{f}_p\)) yields far worse LPIPS than the full configuration with the two cross-attentions, demonstrating that the mixed features generated by cross-stream interaction are critical for the leap in performance.

2. FPSM: Pixel-Wise Hard Selection Dependent on Local Content and Scale Factor

The design motivation behind FPSM is that the optimal choice among the four candidate features varies with spatial locations and scale factors, whereas soft mixing (e.g., weighted averaging/feature modulation) would produce degraded outputs between features from different statistical manifolds. FPSM first predicts local confidence maps \(Conf_f\) and \(Conf_p\) from the fidelity and perceptual feature maps via convolutional layers, and then samples local confidence values in the neighborhood of \(x_q\). The final confidence of each candidate is obtained by a fully connected layer taking the element-wise product of its corresponding attention map and the local confidence values, concatenated with the cell size (which encodes the target scale factor). During training, Gumbel-Softmax is used to achieve differentiable discrete selection (with temperature annealing from 1 to 0.001), while argmax is directly applied during inference. Visual analysis demonstrates that FPSM prioritizes the fidelity feature \(\tilde{f}_f\) in flat areas and the perceptual feature \(\tilde{f}_p\) in texture-rich areas. Meanwhile, the selection ratio of \(\tilde{f}_{fp}\) (fidelity-anchored + perceptual details) increases significantly at high magnification factors—exactly matching the demand that "higher scale factors require more perceptual supplementation."

3. Five-Way Supervision Strategy Ensures Independent Utility of Each Candidate

To ensure that the confidence scores of FPSM truly reflect the differences in reconstruction quality of the candidates rather than artifacts of uneven training, FPLIA imposes independent supervision losses on all five outputs (the selected output + four individual candidates). The selected output uses standard MSE, the fidelity self-attention output also uses MSE, while the losses for the three perception-related outputs (\(\tilde{f}_p, \tilde{f}_{pf}, \tilde{f}_{fp}\)) are weighted by \(\mu/\sigma\)—borrowing the one-step diffusion approximation strategy from Kim et al. to dynamically adjust error weights according to the signal-to-noise ratio of the diffusion process. The five losses are summed directly without manual weighting. This design ensures that even if the soft selection of FPSM is inaccurate in the early stages of training, each candidate branch can still independently learn meaningful representations, providing a reliable foundation for subsequent confidence estimation.

Loss & Training¶

The training loss consists of a five-branch MSE: the selected output \(L_{select}\) and the fidelity output \(L_f\) are standard MSE; the perception-related outputs \(L_p, L_{pf}, L_{fp}\) adopt \(\mu/\sigma\)-weighted MSE (\(\frac{\mu}{\sigma}(I^{HR_i} - I^{GT})\), where \(\mu/\sigma\) reflects the signal-to-noise ratio of the diffusion process). The five losses are summed without weighting. During training, FPAM, FPSM, and the decoder are updated, while SwinIR and Stable Diffusion v1.5 are frozen. The Gumbel-Softmax temperature is annealed from 1 to 0.001 to achieve a smooth transition from soft selection during training to hard selection during inference.

Key Experimental Results¶

Main Results¶

We evaluate integer scale factors and non-integer scale factors from \(\times 4\) to \(\times 16\) on Set5 / Set14 / B100 / Urban100, reporting both PSNR (fidelity) and LPIPS (perceptual quality).

Dataset	Metric	Best Regression Method	Best Diffusion Method	FPLIA	Gain (vs Regression)	Gain (vs Diffusion)
Set14 ×4	PSNR↑	29.10 (HIIF)	27.77 (Kim)	28.48	-2.13%	+2.56%
Set14 ×4	LPIPS↓	0.267 (HIIF)	0.201 (Kim)	0.188	+29.59%	+6.47%
B100 ×8	PSNR↑	25.09 (HIIF)	24.18 (Kim)	24.70	-1.55%	+2.15%
B100 ×8	LPIPS↓	0.532 (CiaoSR)	0.466 (Kim)	0.431	+18.98%	-0.23%
Urban100 ×4	PSNR↑	27.44 (HIIF)	26.42 (Kim)	26.87	-2.08%	+1.70%
Urban100 ×4	LPIPS↓	0.187 (CiaoSR)	0.170 (Kim)	0.160	+14.44%	+5.88%

FPLIA outperforms diffusion methods in almost all settings (gains in both PSNR and LPIPS). Compared to regression methods, it trade-offs about 2-3% in PSNR for a 15-30% improvement in LPIPS, placing it on the Pareto frontier of fidelity and perception. During inference, it adds only <5% latency and <2% GPU memory compared to the Kim et al. baseline.

Ablation Study¶

Configuration	×4 PSNR↑	×4 LPIPS↓	×8 PSNR↑	×8 LPIPS↓	Note
w/o FPAM + w/o FPSM	28.40	0.231	25.15	0.383	Baseline (Direct fusion)
+FPAM but w/o FPSM	28.53	0.188	25.28	0.371	With interaction but no selection
w/o FPAM + FPSM	28.60	0.211	25.24	0.366	With selection but no interaction
Full model	28.91	0.183	25.44	0.338	Synergy gain exceeds individual contributions
Only \(\tilde{f}_f\)	29.51	0.258	25.84	0.414	Highest PSNR, worst LPIPS
Only \(\tilde{f}_{fp}\)	28.28	0.186	25.03	0.352	Fidelity-anchored + perceptual details gives the most balanced performance
\(\tilde{f}_f + \tilde{f}_p\)	28.71	0.206	25.31	0.361	Combining two self-attentions is far inferior to the full configuration
Pixel-wise selection vs Concatenation	28.91 vs 28.53	0.183 vs 0.188	25.44 vs 25.28	0.338 vs 0.371	Hard selection clearly outperforms soft mixing

Key Findings¶

The synergistic gain of FPAM and FPSM exceeds the sum of their individual contributions: cross-attention generates diverse candidates, and the selection mechanism chooses the optimal one—both are indispensable.
Between the two cross-attention features, the independent performance of \(\tilde{f}_{fp}\) (fidelity query \(\times\) perceptual key-value) vastly outperforms \(\tilde{f}_{pf}\) (perceptual query \(\times\) fidelity key-value), indicating that supplementing perceptual details on a fidelity anchor is more effective than the reverse.
The selection pattern of FPSM dynamically changes with the scale factor: at \(\times 2\), \(\tilde{f}_f\) accounts for 47%, which drops to 33% at \(\times 16\), while \(\tilde{f}_{fp}\) rises from 23% to 45%—quantitatively validating that "higher scale factors rely more on perceptual supplementation."
Pixel-wise selection outperforms channel-wise selection: the fidelity-perception trade-off is spatial and does not vary along the channel dimension.

Highlights & Insights¶

The paper reformulates the fidelity-perception trade-off from "choosing a side" to "dual-stream parallel + pixel-wise selection of the best." The framework design precisely matches the structure of the problem—feature diversity comes from asymmetric interaction, and selection is guided by local content and scale factors.
The Asymmetric Bidirectional Cross-Feature Attention (FPAM) elegantly solves the calculation issue where "direct concatenation leads to degradation." Each directional attention generates an independently useful yet mutually irreplaceable hybrid feature, providing meaningful discrete options for subsequent selection.
The Gumbel-Softmax discrete selection strategy coupled with confidence estimation in FPSM uses an annealing temperature to achieve a smooth transition from soft to hard selection during training—preventing degradation caused by soft-mixing features from different statistical manifolds.
The framework is backbone-agnostic and can switch between different fidelity extractors and perceptual generators, showing good transferability.

Limitations & Future Work¶

The perceptual generator currently uses Stable Diffusion v1.5, which still incurs a much higher computational load than pure regression-based approaches. Although the incremental overhead is only 5%, the total latency is about 98 seconds (on an A6000 for \(128 \to 512\)), making it impractical for real-time scenarios.
Confidence estimation in FPSM relies on global confidence maps, which may lead to unstable selection in highly non-unform long-tailed texture regions.
This paper focuses primarily on super-resolution. It is worth exploring whether the core idea of this method (dual-stream feature interaction + pixel-wise selection) can be generalized to other low-level vision tasks (denoising, deblurring, inpainting).
The temperature annealing schedule of Gumbel-Softmax (\(1 \to 0.001\)) requires hyperparameter tuning, which may need readjustment for different datasets or tasks.

vs LIIF / LTE / CiaoSR / HIIF: These regression methods only utilize a single-stream fidelity feature combined with local implicit functions. FPLIA is the first to systematically introduce the perceptual features of diffusion models into the deep learner of ASISR.
vs Kim et al. (2024): This is the closest work, which feeds regression features as conditional inputs to a diffusion pipeline but keeps the two streams loosely coupled. FPLIA designs explicit bidirectional interaction attention (FPAM) and pixel-wise optimal selection (FPSM) instead of simple conditional control.
vs IDM: IDM completely replaces the regression backbone with a diffusion model, generating good textures but lacking structural constraints. FPLIA retains the fidelity stream as a structural anchor and injects structural constraints into texture generation via FPAM's cross-attention.
Transferable Ideas: The design paradigm of "dual-stream asynchronous interaction + pixel-wise hard selection" can be applied to other low-level vision tasks requiring the fusion of heterogeneous features, such as feature fusion across different receptive fields or feature alignment across different modalities.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ It transforms the fidelity-perception trade-off from "siding" into a "generating candidates + picking the best" paradigm. The motivation for FPAM's asymmetric bidirectional interaction is solid and naturally implemented, rather than just stacking modules.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive comparisons are conducted on 5 datasets, across \(\times 4 \sim \times 16\) integer and non-integer factors, against 7 baseline methods. Ablation studies cover every module, branch, and selection strategy, supplemented by visualization of FPSM's selection distribution, forming a complete chain of evidence.
Writing Quality: ⭐⭐⭐⭐ The derivation of motivations and design analysis (Remark 1 + dual-axis analysis of space/scale) are clear. However, the experimental tables are somewhat lengthy (~5 large tables) and could be condensed for non-core results.
Value: ⭐⭐⭐⭐⭐ It resolves the long-standing fidelity-perception dilemma in ASISR, offering a feasible and efficient dual-stream fusion framework. It provides valuable references for future ASISR and more general dual-stream feature fusion designs.