title: >- [Paper Note] Fill2SR: Repurposing Inpainting Diffusion Transformers for Real-World Super-Resolution description: >- [ECCV 2026][Image Generation][Super-Resolution] Fill2SR repurposes the native masked-image slot of an inpainting Diffusion Transformer as a dense pixel-aligned evidence channel under a full-image mask, achieving memory-predictable real-world SR up to QHD without extra spatial control branches. tags: - ECCV 2026 - Image Generation - Real-World Super-Resolution - Diffusion Transformer - Rectified Flow date: 2026-09-19 content_hash: 3ad3ded7703383f8

Fill2SR: Repurposing Inpainting Diffusion Transformers for Real-World Super-Resolution¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Xingfu-Yi/Fill2SR
Area: Image Generation
Keywords: Real-World Super-Resolution, Image Restoration, Inpainting Diffusion Transformers, Rectified Flow Matching, Adapter Tuning

TL;DR¶

Fill2SR dispenses with ControlNet-style auxiliary branches and spatial token streams by repurposing the native masked-image slot of an inpainting DiT into a dense low-quality evidence channel under a full-image mask, enabling memory-predictable mixed-resolution training up to native QHD on a single 32GB GPU.

Background & Motivation¶

Real-world image super-resolution (SR) and restoration aim to recover high-quality (HQ) images from low-quality (LQ) observations degraded by complex mixtures of blur, noise, downsampling, and compression. Traditional discriminative restorers such as SwinIR and Restormer are governed by the perception-distortion trade-off, inevitably yielding over-smoothed, plasticky textures under severe real-world corruptions. Generative diffusion models have consequently emerged as the dominant foundation for photo-realistic restoration. However, conventional text-to-image (T2I) backbones are pre-trained for unconstrained open-domain synthesis and lack native pixel-level spatial alignment priors, making them prone to structural distortion and geometric hallucinations under severe blind degradations.

To enforce fidelity, existing approaches typically graft ControlNet-style auxiliary branches onto the backbone or inject dense spatial conditioning token streams directly into DiT attention layers. While effective to some extent, this paradigm suffers from two structural bottlenecks. First, memory and compute overheads induced by external control branches scale superlinearly (often quadratically with spatial tokens) as resolution grows, severely limiting training resolutions. Second, bound by VRAM ceilings, these control modules are predominantly trained at a single fixed resolution (e.g., $1024^2$), binding the model to a rigid resolution-specific prior that collapses into hallucinations when evaluated on arbitrary scales or multi-aspect-ratio inputs.

To overcome these structural limitations, this paper advocates building restoration on an interface that is natively pixel-aligned: mask-conditioned inpainting Diffusion Transformers. The core idea is to repurpose the native masked-image slot of an inpainting DiT as a dense pixel-aligned evidence channel under a full-image mask, eliminating auxiliary spatial branches and enabling stable mixed-resolution training up to native QHD via reverse-degradation rectified flow and offline reference-conditioned degradation transfer.

Method¶

Overall Architecture¶

Fill2SR reformulates real-world super-resolution directly within the native interface of an inpainting DiT (FLUX.1-Fill-dev). Given an LQ image $Y \in \mathbb{R}^{3 \times \frac{H}{s} \times \frac{W}{s}}$, it is first bicubic-upsampled to the target spatial dimension and encoded into the VAE latent space as $y \in \mathbb{R}^{c \times h \times w}$. Instead of attaching external side networks, Fill2SR injects $y$ directly into the native masked-image slot while enforcing a constant full-image mask $m \equiv 1$. Paired with compact fixed-length visual-semantic tokens from a frozen Redux embedder, the inputs are concatenated channel-wise as $(z_t, y, m)$, preserving spatial correspondences without token inflation. A parameter-efficient LoRA adapter is trained on attention and feed-forward projections to learn a reverse-degradation conditional rectified flow that transports pure Gaussian noise $z_1$ to the clean HQ latent $x$. Furthermore, an offline Reference-Conditioned Degradation Transfer (RCDT) pipeline synthesizes realistic training pairs using frozen multi-modal models, introducing zero inference overhead.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    LQ["Low-Quality Image Y (Upsampled & VAE-encoded to y)"] --> IIEA["IIEA Evidence Adapter<br/>Full-image mask m≡1 activates native inpainting slot"]
    LQ --> Redux["Lightweight Visual-Semantic Guidance<br/>Redux extracts 768 fixed-length semantic tokens"]
    IIEA --> DiT["Reverse-Degradation Rectified Flow Backbone (FLUX-Fill + LoRA)<br/>Channel-concatenated (zt, y, m) eliminates external branches"]
    Redux --> DiT
    RCDT["Offline Degradation Transfer RCDT<br/>Qwen3-VL distills descriptors + editor synthesizes pairs"] -.->|Mixed in final 20% training steps| DiT
    DiT --> HQ["Generated HQ Latent z0 → VAE Decoder to Output Image"]

Key Designs¶

1. IIEA (Inpainting-Interface Evidence Adapter): Native Slot Repurposing Eliminating Branch Overheads

Standard diffusion SR frameworks introduce auxiliary ControlNet copies or spatial token streams, causing memory demands to explode at high resolutions. Fill2SR observes that FLUX-Fill natively consumes a channel-concatenated latent tuple $(z_t, z^{\text{masked}}, m)$. In standard inpainting, pixels under $m=0$ represent preserved context enforced to remain unchanged, whereas $m=1$ designates target completion regions. Under a naive full-image mask $m \equiv 1$, the native masked-image latent $z^{\text{masked}}$ collapses to an all-zero vector $x'$, discarding all input observation. IIEA resolves this by fixing $m \equiv 1$ across the entire image—disabling rigid verbatim copy-preserve semantics since every pixel in SR requires degradation removal and super-resolution refinement—while populating the native $z^{\text{masked}}$ slot with the upsampled LQ latent $y$: $$c_{\mathrm{IIEA}}(y; \alpha) = [y(\alpha), \mathbf{1}]$$ where $\alpha \in [0, 1]$ represents an optional pixel-space evidence scaling factor. Because conditioning information is passed entirely through channel concatenation in latent space, the token sequence length remains strictly constant, completely avoiding quadratic token growth and natively supporting mixed-resolution training from $512$ to $2048$ pixels.

2. Reverse-Degradation Conditional Rectified Flow: Full-Image Endpoint Trajectory Modeling

While inpainting backbones are pre-trained for local context-conditioned infilling, blind super-resolution requires learning a trajectory from full-image corrupted observations to pristine HQ targets. Fill2SR formulates this task using Rectified Flow Matching (RFM). Assuming an implicit degradation operator $\mathcal{T}$ such that $y = \mathcal{T}(x)$, a linear interpolation trajectory is established between pure Gaussian noise $z_1 \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and the target HQ latent $x$: $$z_t = (1 - t)x + t z_1, \quad t \sim \mathrm{Unif}(0, 1)$$ The DiT velocity field $v_\theta(z_t, \mathcal{T}(x), t)$ is trained via the RFM objective: $$\mathcal{L}_{\mathrm{RFM}} = \mathbb{E}_{t, x, z_1} \left\| v_\theta(z_t, \mathcal{T}(x), t) - (z_1 - x) \right\|_2^2$$ At inference time, the model executes full generation from pure noise $z_1$ down to $t=0$. Empirical analysis demonstrates that full generation adheres strictly to the learned flow manifold, whereas partial latent refinement starting from intermediate timesteps ($\beta < 1$) deviates from the manifold trajectory and causes severe perceptual degradation.

3. Lightweight Visual-Semantic Guidance: Fixed-Length Tokens Preventing Category Drift

Severe real-world corruptions often obliterate fine details and warp local low-frequency contours, predisposing generative backbones to semantic hallucinations. Instead of employing a heavyweight vision-language model (VLM) at test time to produce unstructured text descriptions, Fill2SR adopts a frozen FLUX-Redux image embedder. Regardless of input aspect ratio or resolution, Redux compresses the input image $y$ into a fixed-length sequence of $L = 768$ semantic tokens $s(y) = \mathrm{Redux}(y) \in \mathbb{R}^{L \times D}$. These tokens are fed through the native cross-attention context pathway, providing global semantic anchoring with strictly bounded computational cost.

4. RCDT (Reference-Conditioned Degradation Transfer): Offline Multi-Modal Degradation Transfer

Because paired real-world training samples are non-existent and analytical degradation models fail to capture in-the-wild sensor statistics, RCDT introduces an offline data-synthesis pipeline without incurring test-time compute: 1. Degradation Descriptor Distillation: A frozen multi-modal model (Qwen3-VL) inspects an unpaired real-LQ image pool $\{y^{\mathrm{real}}_i\}$ to extract physical corruption descriptions $d_i$, forming a prompt pool $\mathcal{P}$; 2. Content-Aware Specification: For any clean HQ training image $x$, a sampled degradation descriptor $d \sim \mathrm{Unif}(\mathcal{P})$ is contextualized by Qwen3-VL into a content-specific prompt $\tilde{d} = \mathrm{Qwen3VL}(x, d)$; 3. Instruction-Based Degradation Transfer: A frozen instruction image editor (Qwen-Image-Edit) applies the described degradation to $x$, producing a photorealistic paired LQ observation $\hat{y} = \mathrm{Edit}(x; \tilde{d})$. To prevent semantic distortion, RCDT enforces degradation-only prompts and VLM-based geometric verification. In training, RCDT pairs are introduced exclusively in the final 20% of training steps (10k pairs) following 300k synthetic pair pre-training, effectively achieving domain alignment while retaining synthetic structural fidelity.

Loss & Training¶

Fill2SR is fine-tuned parameter-efficiently using LoRA (rank 256) on attention and FFN projections of the frozen 12B FLUX-Fill backbone, updating only 0.65B parameters while freezing the backbone, VAE, and Redux modules. The model is trained for 20k steps with a learning rate of $2.5 \times 10^{-4}$ (including 2.5k warmup steps) and a global batch size of 16. Benefiting from the bounded memory footprint of IIEA, training leverages mixed-resolution and mixed-aspect-ratio bucketing: 512–1536 pixels in the early stage and 1024–2048 pixels in the later stage, up to native QHD ($2560 \times 1440$). Total training requires only 4× RTX 5880 Ada (48GB) for 14 days (50.6 A100-equivalent GPU-days).

Key Experimental Results¶

Main Results¶

The model is thoroughly evaluated on synthetic benchmarks (DIV2K, LSDIR, FFHQ averaged across D1–D3 degradation levels) and real-world benchmarks (RealPhoto60 and RealLQ250 across 2×, 4×, and 8× scales).

Synthetic Benchmark Evaluation (Table 1, Average of D1–D3)

Dataset	Metric	Real-ESRGAN	SeeSR	SUPIR	FaithDiff	Ours-Base (w/o RCDT)	Ours-Full (+RCDT)
DIV2K_VAL	PSNR↑	23.3520	22.5903	22.6610	22.4013	22.4596	21.7831
	SSIM↑	0.6414	0.5807	0.5748	0.5574	0.5872	0.5679
	LPIPS↓	0.4017	0.4165	0.4032	0.4051	0.3866	0.3981
	MUSIQ↑	58.9699	69.0464	64.6142	68.1122	65.9644	67.8893
	CLIPIQA↑	0.5252	0.6015	0.5858	0.6184	0.6173	0.6271
LSDIR_VAL	PSNR↑	20.7295	20.0585	19.9397	19.6345	19.6930	19.2354
	SSIM↑	0.5670	0.4900	0.4847	0.4622	0.5019	0.4890
	LPIPS↓	0.4055	0.4126	0.4202	0.4083	0.3878	0.3974
	MUSIQ↑	64.5601	72.6831	67.9508	71.6307	72.3820	73.1206
	CLIPIQA↑	0.5674	0.6388	0.6234	0.6653	0.6840	0.6859

Real-World Benchmark Evaluation (Table 2)

Dataset & Scale	Metric	Real-ESRGAN	SeeSR	SUPIR	DiT4SR	Ours-Base	Ours-Full
RealPhoto60 (2×)	MUSIQ↑	59.0296	71.8052	69.6326	72.5731	70.6389	73.5949
	MANIQA↑	0.4797	0.6079	0.6116	0.6309	0.6307	0.6468
	CLIPIQA↑	0.5068	0.5959	0.5972	0.6192	0.6780	0.7137
RealPhoto60 (4×)	MUSIQ↑	50.9528	58.1401	57.0462	63.7202	61.8281	65.8413
	MANIQA↑	0.4785	0.5530	0.5269	0.5803	0.5957	0.6042
	CLIPIQA↑	0.4247	0.5386	0.4542	0.5879	0.5207	0.5486
RealLQ250 (4×)	MUSIQ↑	62.5154	70.3728	63.2153	70.5121	62.1914	71.5498
	MANIQA↑	0.5239	0.5927	0.5812	0.6154	0.5751	0.6387
	CLIPIQA↑	0.4359	0.5568	0.4542	0.5617	0.5384	0.6287
RealLQ250 (8×)	MUSIQ↑	41.9557	58.9324	51.3011	61.8378	50.0516	59.3031
	MANIQA↑	0.4582	0.5509	0.5173	0.5766	0.5516	0.5876
	CLIPIQA↑	0.3614	0.5071	0.3849	0.5403	0.4293	0.4784

Ablation Study¶

Ablation on Core Components and Hyper-parameters (Table 5, D1–D3 Average)

Configuration	DIV2K PSNR↑	DIV2K LPIPS↓	LSDIR PSNR↑	LSDIR LPIPS↓	RealLQ250 CLIPIQA↑	RealLQ250 MANIQA↑	RealLQ250 MUSIQ↑
Default (Full Model)	21.7831	0.3981	19.2354	0.3974	0.6076	0.6375	67.7982
w/o RCDT Data	22.4596	0.3866	19.6930	0.3878	0.5350	0.5907	59.9384
w/o Redux Tokens	21.1379	0.4125	18.6203	0.4118	0.5963	0.6328	67.1587
Redux (Zero Image)	21.4694	0.4067	18.9790	0.4081	0.6023	0.6336	67.4110
Redux (Noise Image)	21.6229	0.4047	19.0723	0.4043	0.6059	0.6355	67.4399
Evidence $\alpha=0.5$	20.9871	0.4105	18.5959	0.4040	0.6031	0.6333	67.5953
Evidence $\alpha=0.7$	21.4020	0.4042	18.9017	0.4004	0.6037	0.6348	67.5593
Evidence $\alpha=0.9$	21.7308	0.3994	19.1563	0.3981	0.6058	0.6364	67.6596
Latent Refinement $\beta=0.5$	23.7536	0.5568	21.1851	0.5731	0.3047	0.3987	28.2379
Latent Refinement $\beta=0.7$	23.8974	0.5261	21.2508	0.5501	0.3075	0.3972	28.8641

Key Findings¶

RCDT Domain-Transfer Trade-off: Removing RCDT training data improves synthetic LPIPS from 0.3981 to 0.3866 on DIV2K, but causes a catastrophic collapse in real-world no-reference quality on RealLQ250 (MUSIQ drops from 67.7982 to 59.9384). This verifies that synthetic degradation introduces domain bias, while RCDT aligns the flow toward genuine in-the-wild statistics.
Failure of Partial Latent Refinement: Initializing trajectories from intermediate latents ($\beta=0.5$) drastically increases PSNR to 23.7536 due to high proximity to blurry inputs, but perceptual metrics severely degrade (LPIPS spikes to 0.5568, MUSIQ plummets to 28.2379), confirming that intermediate initializations fall off the learned rectified flow manifold.
Hardware Efficiency and Predictability (Table 4): At $1536^2$ resolution, competing frameworks (SUPIR, DreamClear, FaithDiff) incur out-of-memory (OOM) errors on a 32GB GPU. Fill2SR operates comfortably on a single 32GB GPU thanks to the zero-branch IIEA design, requiring only 50.6 A100e days for training and 18.1 A100e days for offline data synthesis.

Highlights & Insights¶

Inpainting Interface as a Native Restoration Anchor: Repurposing FLUX-Fill's masked-image slot under $m \equiv 1$ replaces memory-heavy ControlNets with zero-overhead channel concatenation, demonstrating that native inpainting priors offer superior spatial alignment over unconstrained T2I priors.
Decoupled Offline Degradation Modeling: Offloading real-world degradation perception to offline multi-modal agents (Qwen3-VL + Qwen-Image-Edit) bypasses complex online degradation estimators, providing high-fidelity supervision without adding inference latency.
Resolution-Agile DiT Restoration: Pairing fixed-length Redux tokens with channel-level latent concatenation completely decouples conditioning costs from image resolution, unlocking robust mixed-resolution training up to native QHD ($2560 \times 1440$).

Limitations & Future Work¶

Generative Hallucinations Under Severe Degradations: In scenarios with extreme information loss, the model synthesizes plausible yet unverified details; it is unsuitable for forensics, clinical diagnostics, or legal authentication where pixel-exact truth is mandatory.
Inference Latency of Large Backbones: While only 0.65B LoRA parameters are trained, inference requires running iterative numerical integration across a 12B FLUX-Fill backbone, which remains significantly slower than single-step feed-forward regressors.
Potential Geometric Drift in Synthesized Pairs: Although filtered by VLM checks, instruction-based image editors may introduce minor sub-pixel geometric shifts in RCDT training pairs.

vs ControlNet / SUPIR / SeeSR (T2I Adaptation): Prior works attach auxiliary structural branches or inject large spatial token sequences, inducing superlinear memory growth and restricting training to fixed resolutions; Fill2SR utilizes the native inpainting slot with zero extra branches, running predictably on single 32GB GPUs up to QHD.
vs DiT4SR / DreamClear (DiT-Based SR): DreamClear requires intricate dataset curation and degradation prediction models, while DiT4SR relies on specialized attention adaptations; Fill2SR keeps the core architecture pristine through LoRA on FLUX-Fill, transferring complexity into offline RCDT data generation.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneering reuse of inpainting interfaces for branch-free super-resolution]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive synthetic and real-world benchmarks, rigorous ablations, and cross-hardware A100e profiling]
Writing Quality: ⭐⭐⭐⭐⭐ [Sharp problem formulation, disciplined technical exposition, and thorough empirical transparency]
Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly scalable blueprint for deploying foundation DiTs to extreme-resolution restoration]