ReAL: Reference-to-Image (R2I) Aware Latent Diffusion for Image Super-Resolution¶

Conference: ECCV 2026
Paper: ECCV 2026
Area: Image Generation
Keywords: Reference-based Super-Resolution, Retrieval-Augmented Generation, Latent Diffusion Model, Neural Hashing, Key-Value Cache Fusion

TL;DR¶

ReAL departs from the conventional dependence on text-to-image (T2I) priors and classifier-free guidance (CFG) in super-resolution, establishing a pure reference-to-image (R2I) latent diffusion framework with one-time reference key-value caching and direct self-attention injection, drastically cutting latency and memory while eliminating semantic drift.

Background & Motivation¶

In real-world single image super-resolution (Real-ISR), input low-resolution (LR) observations suffer from severe, unknown physical degradations. The extreme loss of high-frequency textures turns reconstruction into an ill-posed inverse problem. While early generative adversarial network (GAN) approaches restored sharp edges, they frequently suffered from training instability and unnatural structural artifacts. Recently, latent diffusion models (LDMs) leveraging large-scale text-to-image (T2I) generative priors have become dominant, often paired with vision-language models (VLMs) to automatically generate descriptive text prompts. However, under heavy degradation, the likelihood constraint from the low-resolution observation is extremely broad and weakly informative. Consequently, the diffusion trajectory becomes governed primarily by the abstract textual prior, causing the model to hallucinate semantically inconsistent details—such as transmuting a blurred leaf patch into a human face—and inducing severe semantic drift across text signs, patterns, and boundaries.

Reference-based super-resolution (RefSR) aims to ground generation in verifiable visual evidence by supplying an auxiliary high-resolution reference image. Recent retrieval-augmented diffusion methods like CoSeR and iRAG retrieve exemplars from external databases, yet they fundamentally adapt frozen T2I backbones, injecting reference cues through intermediate textual mappings, cross-attention adapters, or abstract semantic projections. This indirect conditioning dilutes concrete high-frequency visual cues and inherits the rigid dependence on classifier-free guidance (CFG), which doubles the number of network function evaluations and results in prohibitive computational and memory overhead.

This paper addresses this bottleneck by fundamentally discarding text-driven generation in favor of a pure reference-to-image (R2I) paradigm driven strictly by visual blueprints. Core idea: eliminate all text priors and classifier-free guidance, encode the retrieved reference image only once into a self-attention key-value cache, and inject these cached features directly into the LR-anchored self-attention layers during denoising to enable efficient, hallucination-resistant texture transfer.

Method¶

Overall Architecture¶

The overall pipeline of ReAL samples directly from the posterior distribution \(p_\phi(I_{\text{SR}} \mid I_{\text{LR}}, I_{\text{ref}})\) through three tightly coordinated components: neural hash-based retrieval, one-time forward feature caching, and self-attention key-value fusion. First, an LR query is mapped to a compact binary code to retrieve the most visually and semantically correlated HR exemplar from an external database. Second, the retrieved reference image is passed through the denoising U-Net exactly once at the beginning of inference to extract and cache its Key and Value tensors across all self-attention layers. Finally, throughout the iterative denoising trajectory, the LR latent serves as a channel-concatenated structural anchor while the cached reference features are concatenated along the token dimension to guide detail reconstruction without classifier-free guidance.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Low-Resolution Input $I_{LR}$"] --> B["Neural Hashing Cross-Modal Retrieval<br/>Contrastive InfoNCE maps query to binary code"]
    B --> C["External Reference Database Search<br/>Nearest-neighbor matching yields optimal $I_{ref}$"]
    C --> D["One-Time Forward Feature Caching<br/>Single pass extracts and caches $K_{ref}, V_{ref}$"]
    D --> E["Self-Attention Key-Value Fusion<br/>LR query anchored; tokens concatenated along sequence"]
    A -->|VAE encode $z_{LR}$ channel concat| E
    E --> F["CFG-Free Pure R2I Diffusion Denoising<br/>Single evaluation per timestep"]
    F --> G["Super-Resolved Output $I_{SR}$"]

Key Designs¶

1. Neural Hashing Cross-Modal Retrieval: millisecond nearest-neighbor search in learned embedding space Conventional feature matching over dense high-dimensional deep descriptors (such as VGG embeddings) imposes massive latency overheads that do not scale to large reference repositories. ReAL integrates a compact binary neural hashing module optimized via contrastive learning. Using a frozen VGG backbone followed by a learnable hashing head, images are projected into a shared binary Hamming space. During training, two augmented views are constructed per sample and aligned using a symmetric InfoNCE objective:

\[ \mathcal{L}_{k,1} = -\log \frac{\exp(\mathcal{C}(\mathbf{b}_{k,1}, \mathbf{b}_{k,2}) / \tau)}{\sum_{\ell=1}^N \sum_{j \in \{1,2\}} \mathbb{I}_{(\ell,j) \neq (k,1)} \exp(\mathcal{C}(\mathbf{b}_{k,1}, \mathbf{b}_{\ell,j}) / \tau)} \]

At inference time, mapping the LR query into a compact binary code enables nearest-neighbor retrieval via fast Hamming distance calculations across the candidate database. In benchmark profiling, hash retrieval executes in just 62 ms compared to 3830 ms for dense VGG matching, rendering exemplar retrieval overhead negligible.

2. One-Time Forward Feature Caching: decoupling reference encoding from iterative denoising In multi-step diffusion processes, re-encoding the reference image across dozens of timesteps introduces prohibitive computational waste. ReAL introduces a Reference Fusion Module (RFM) inspired by KV caching architectures. Prior to the iterative reverse diffusion loop, the high-resolution reference \(I_{\text{ref}}\) is processed through the denoising U-Net in a single forward pass. The resulting Key (\(K_{\text{ref}}\)) and Value (\(V_{\text{ref}}\)) tensors from every self-attention layer are cached into GPU memory. Because the clean reference is uncorrupted by diffusion timesteps, its structural and high-frequency textural representations remain constant throughout sampling, serving as an immutable, plug-and-play visual blueprint.

3. Self-Attention Key-Value Fusion: query-anchored feature borrowing with natural hallucination resistance During reverse diffusion, the U-Net operates on the channel-wise concatenation of the noisy latent \(z_t\) and the encoded LR latent \(z_{\text{LR}} = \mathcal{E}(I_{\text{LR}})\). Within each self-attention block, the network computes the main path query (\(Q_{\text{main}}\)), key (\(K_{\text{main}}\)), and value (\(V_{\text{main}}\)). The RFM concatenates the cached reference tensors along the token sequence dimension:

\[ K_{\text{fused}} = [K_{\text{main}}, K_{\text{ref}}], \quad V_{\text{fused}} = [V_{\text{main}}, V_{\text{ref}}] \]

Attention is subsequently evaluated as \(\text{Attention}(Q_{\text{main}}, K_{\text{fused}}, V_{\text{fused}})\). Crucially, the reference is injected exclusively on the key and value sides while the query matrix \(Q_{\text{main}}\) remains anchored to the LR latent. This design preserves the spatial geometry of the degraded input while allowing each spatial token to selectively borrow relevant high-frequency details. When an irrelevant or cross-domain exemplar is supplied, the unified softmax distribution naturally drives the attention weights on the reference tokens close to zero, smoothly falling back to LR-only reconstruction without requiring explicit gating networks.

Loss & Training¶

The denoising U-Net in ReAL is initialized from pre-trained Stable Diffusion 2-base weights, while the VAE encoder-decoder remains entirely frozen throughout training. Because the retrieved reference directly supplies explicit fine-grained textures, the framework dispenses with unstable GAN adversarial and perceptual losses, optimizing solely with the standard latent-space \(\ell_2\) noise prediction objective:

\[ \mathcal{L} = \mathbb{E}_{z_0, \epsilon, t, c} \left[ \|\epsilon - \epsilon_\theta(z_t, t, c)\|^2 \right] \]

where the condition \(c = (t, z_{\text{LR}}, \{K_{\text{ref}}, V_{\text{ref}}\})\) contains no text token embeddings. At test time, sampling is performed using a 50-step DDIM schedule. Because there is no null-text condition to evaluate, classifier-free guidance (CFG) is omitted entirely, cutting the required network evaluations by 50% relative to standard T2I diffusion models.

Key Experimental Results¶

Main Results¶

ReAL is evaluated across three real-world super-resolution benchmarks (DIV2K validation, RealSR, and DRealSR), measuring fidelity (PSNR, SSIM, LPIPS) and perceptual quality (CLIP-IQA+, MUSIQ). All competing diffusion baselines are trained on the unified DF2K and OST dataset.

Dataset	Metric	ESRGAN	BSRGAN	LDM	StableSR	PASD	DifBIR	SeeSR	FaithDiff	CoSeR	iRAG	Ours (ReAL)
DIV2K Valid	PSNR↑	20.067	19.858	18.727	18.956	19.453	18.747	19.157	18.375	19.579	19.280	19.650
	SSIM↑	0.525	0.504	0.438	0.435	0.497	0.418	0.484	0.443	0.498	0.458	0.507
	LPIPS↓	0.395	0.422	0.428	0.415	0.462	0.451	0.392	0.433	0.415	0.410	0.391
	CLIP-IQA+↑	0.603	0.559	0.541	0.628	0.625	0.727	0.710	0.652	0.599	0.676	0.646
	MUSIQ↑	58.397	58.199	58.743	62.083	60.877	69.379	68.594	69.316	56.543	66.480	63.325
RealSR	PSNR↑	21.401	21.189	20.338	20.263	20.706	20.132	20.609	18.986	21.007	20.860	21.044
	SSIM↑	0.616	0.594	0.516	0.486	0.593	0.486	0.571	0.508	0.587	0.551	0.608
	LPIPS↓	0.423	0.431	0.464	0.481	0.410	0.487	0.401	0.424	0.407	0.372	0.375
	CLIP-IQA+↑	0.601	0.565	0.513	0.624	0.628	0.728	0.711	0.653	0.596	0.666	0.667
	MUSIQ↑	58.447	59.657	56.391	62.072	63.758	69.263	70.613	70.372	56.745	70.280	66.529
DRealSR	PSNR↑	26.010	25.352	23.518	23.397	24.898	22.507	23.999	21.988	24.734	24.080	24.836
	SSIM↑	0.757	0.723	0.592	0.551	0.731	0.500	0.692	0.535	0.698	0.720	0.725
	LPIPS↓	0.350	0.377	0.466	0.507	0.388	0.579	0.390	0.498	0.395	0.367	0.359
	CLIP-IQA+↑	0.557	0.528	0.482	0.580	0.579	0.689	0.672	0.609	0.550	0.621	0.612
	MUSIQ↑	51.285	53.040	51.938	57.299	57.112	65.606	64.829	65.900	52.345	64.630	60.368

Ablation Study¶

Ablation experiments confirm the contribution of each module and demonstrate the system's sensitivity to reference relevance:

Config	PSNR↑	SSIM↑	LPIPS↓	CLIP-IQA+↑	MUSIQ↑	Note
ReAL (full)	27.352	0.803	0.246	0.625	61.565	full model, optimal balance of fidelity and perception
ReAL (-L)	26.689	0.774	0.309	0.608	60.544	removes LR latent concatenation; structural fidelity drops sharply
ReAL (-R)	27.084	0.799	0.250	0.614	60.294	removes reference conditioning; detail and perceptual metrics drop
ReAL (-F)	26.837	0.782	0.248	0.633	61.802	disables RFM caching/fusion; higher perception but degraded fidelity
ReAL (-P)	27.385	0.780	0.263	0.602	60.250	uses random reference image; falls back gracefully to LR baseline

Furthermore, profiling sampling latency and memory demands reveals substantial efficiency improvements:

Method	Sampling Steps	Uses Reference	Latency (s)↓	GPU Memory (GB)↓
DifBIR	50	No	5.06	9.10
SeeSR	50	No	3.23	9.24
CoSeR	50	Yes	16.85	14.94
iRAG	200	Yes	22.35	22.48
ReAL (Ours)	50	Yes	2.65	5.83

Key Findings¶

Superior structural fidelity among generative models: ReAL achieves the best distortion scores (PSNR, SSIM, LPIPS) among diffusion-based super-resolution methods across all three benchmarks. On DIV2K, it outperforms iRAG by 0.37 dB in PSNR and achieves lower LPIPS (0.359 vs 0.395) on DRealSR without relying on hallucination-prone VLM text prompts.
Drastic reduction in runtime latency and memory: By removing classifier-free guidance and encoding the reference once into a KV cache, ReAL requires only 2.65 seconds and 5.83 GB of GPU memory for 50-step inference. This represents an ~85% reduction in latency compared to CoSeR (16.85s) and iRAG (22.35s).
Inherent resilience to irrelevant exemplars: Attention distribution and PCA visualizations show that when presented with mismatched or random exemplars, the query-anchored attention weights assigned to reference tokens naturally collapse toward zero, preventing artificial texture corruption and gracefully preserving the LR baseline structure.

Highlights & Insights¶

Paradigm shift from T2I to pure R2I: Identifies the fundamental mathematical flaw of T2I super-resolution—where a wide likelihood under severe degradation allows broad text priors to overpower low-level evidence—and replaces ambiguous text guidance with deterministic visual exemplars.
One-time KV cache borrowing in diffusion attention: Bridges LLM KV-cache principles into diffusion self-attention, eliminating redundant reference passes across diffusion timesteps.
Query-anchored key-value injection: By fixing the query to the degraded observation while appending reference tokens solely to keys and values, ReAL guarantees that the spatial layout is strictly preserved while external textures are seamlessly integrated.

Limitations & Future Work¶

Higher initial end-to-end retraining overhead: Unlike adapter-based methods that leave Stable Diffusion backbones frozen, ReAL retrains the U-Net denoiser and reference embedding jointly, requiring substantial training compute.
Degradation boundary under extreme blur: Under catastrophic physical blur where structural boundaries vanish entirely from the LR query, the attention mechanism struggles to anchor itself, resulting in overly smoothed outputs despite high-quality reference inputs.
Cross-domain texture leakage: While the softmax fallback suppresses most irrelevant tokens, subtle foreign textural artifacts occasionally bleed into smooth, flat regions when using cross-domain reference images. Future extensions could incorporate multi-reference ranking or confidence-gated adaptive fusion.

vs CoSeR & iRAG: While both adopt reference images, they rely on frozen T2I backbones, intermediate textual/semantic mappings, and expensive CFG sampling; ReAL removes the text branch entirely, using direct KV self-attention concatenation to achieve higher fidelity at a fraction of the inference latency.
vs SeeSR & FaithDiff: These generic diffusion models employ VLMs (e.g., RAM, LLaVA) to generate text prompts, which frequently trigger semantic drift and hallucinations when text descriptions conflict with subtle visual details; ReAL demonstrates that direct visual references provide a substantially more reliable prior than language embeddings for image restoration.

Rating¶

Novelty: ⭐⭐⭐⭐ [Replaces dominant T2I paradigm with a pure R2I KV-cached diffusion model]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across three benchmarks, comprehensive ablations, retrieval latency profiling, and attention mechanism analyses]
Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous probabilistic motivation and clear architectural exposition]
Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly efficient, and hallucination-resistant blueprint for real-world diffusion super-resolution]