Tiled Prompts: Overcoming Prompt Misguidance in Image and Video Super-Resolution

Conference: ECCV 2026
Paper: ECCV Official
Project: https://bryanswkim.github.io/tiled-prompts/
Area: Image Generation
Keywords: Single Image Super-Resolution, Video Super-Resolution, Diffusion Models, Vision-Language Models, Tiled Prompts

TL;DR

Addressing the prompt misguidance issue in latent tiling super-resolution caused by broadcasting a single coarse global prompt, Tiled Prompts extracts tile-specific localized prompts via vision-language models, theoretically lowering the information-theoretic misguidance bound and substantially boosting perceptual fidelity with minimal compute overhead.

Background & Motivation

Generative diffusion and flow-based models have established a new state of the art in Single Image Super-Resolution (SISR) and Video Super-Resolution (VSR) by leveraging rich textual priors as semantic anchors. However, as real-world restoration scales to extreme high resolutions such as 2K, 4K, and 8K, GPU memory constraints make full-resolution latent processing intractable. Consequently, modern inference pipelines standardly rely on latent tiling, where the low-resolution input is partitioned into overlapping spatial grids or spatio-temporal blocks, denoised independently at the tile level, and blended via Gaussian weighting.

Under this latent tiling paradigm, existing methods universally adopt a naive practice: extracting a single global caption describing the entire scene and broadcasting it identically to every local tile. This creates a severe mismatch known as prompt misguidance, which manifests in two destructive failure modes. The first is errors of commission, where global prompts contain descriptions of foreign visual concepts (e.g., conditioning a pavement tile on "blue sky"), which misdirects classifier-free guidance (CFG) gradients into hallucinating irrelevant artifacts. The second is errors of omission, where a coarse global caption completely ignores high-frequency local structures such as street sign typography or architectural textures; lacking specific semantic anchors in an ill-posed inverse problem, the local tile denoiser fails to reconstruct crisp, accurate details. In video super-resolution, this breakdown is aggravated into dual spatial and temporal misguidance, where a static global caption cannot track local dynamic motions across frames.

Fine-tuning foundation super-resolution models on arbitrary multi-resolution splits is prohibitively expensive and lacks flexibility. The core idea of this paper is: instead of broadcasting a one-size-fits-all global caption, leverage a lightweight vision-language model (VLM) to generate dense, context-aware localized prompts for each individual tile, directly suppressing the information-theoretic mutual information gap and bounding the misguidance trajectory without any model retraining.

Method

Overall Architecture

Tiled Prompts establishes a unified, training-free framework for both image and video super-resolution under latent tiling. The input low-resolution image or video sequence is first partitioned into a regular grid of overlapping latent tiles (spatial patches for SISR, spatio-temporal volumes for VSR). Before initiating the reverse diffusion sampling loop, a pre-trained VLM extracts a highly relevant, localized prompt for each individual tile. During each reverse diffusion step, the denoiser evaluates noise/velocity predictions for each tile conditioned on its dedicated prompt and local latent features, accumulating the results into a global canvas weighted by a Gaussian window. The fused latent is finally decoded into the clean high-resolution output.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Low-Resolution Input<br/>Image or Video Sequence"] --> B["Latent Spatio-Temporal Tiling<br/>Overlapping Grid Partitioning"]
    B --> C["Adaptive Local Prompt Extraction<br/>Tile-Specific VLM Descriptions"]
    C --> D["Local Text-Conditioned Denoising<br/>CFG Trajectory Score Guidance"]
    D --> E["Gaussian Weighted Blending<br/>Seamless Latent Canvas Fusion"]
    E --> F["High-Resolution Output<br/>Sharp Reconstructed Media"]

Key Designs

1. Information-Theoretic Formulation of Prompt Misguidance: Bounding Trajectory Divergence To formally diagnose why global prompts degrade local tile reconstruction, the authors formulate the deviation between an ideal localized text prompt \(c^* \sim p^*(c^*|x_L^{(i)})\) and an arbitrary prompt \(c\) through the score-based guidance vector \(\delta_i(c) := \Delta_t^{(i)}(c) - \Delta_t^{(i)}(c^*)\). Under classifier-free guidance scale \(s\), this term shifts the reverse drift by \(s\delta_i(c)\). Across \(x_0\)-prediction, \(\epsilon\)-prediction, and flow-based \(v\)-prediction, Lemma 1 shows that this misguidance is universally governed by score discrepancies: $$ \delta_i(c) = w(t) \left( \nabla_{x_t^{(i)}} \log p_{\theta,t}(x_t^{(i)} \mid x_L^{(i)}, c) - \nabla_{x_t^{(i)}} \log p_{\theta,t}(x_t^{(i)} \mid x_L^{(i)}, c^*) \right) $$ The authors establish an explicit connection between the mutual information gap \(\Delta I := I(\hat{x}_H^{(i)}; c^* \mid x_L^{(i)}) - I(\hat{x}_H^{(i)}; c \mid x_L^{(i)})\) and the cumulative trajectory error. Specifically, Proposition 1 proves that \(\Delta I\) matches the posterior KL divergence and serves as a strict lower bound on the integrated misguidance error: $$ \Delta I \le \frac{1}{2} \int_0^T \lambda(t) \mathbb{E} \left[ |\delta_i(c)(t)|^2 \right] dt $$ Because tile-specific prompts \(c_{\mathrm{local}}^{(i)}\) strictly capture local visual evidence, Proposition 2 proves \(\Delta I_\ell \le \Delta I_g\). This theoretically ensures that using tiled prompts lowers the mathematical floor of the misguidance vector, eliminating both errors of commission and omission during reverse sampling.

2. Tile-Specific Prompt Extraction for Image Super-Resolution: Fine-Grained Semantic Anchoring For image super-resolution, local crops represent fine-grained spatial sub-regions where global captions lose specificity. The framework utilizes a pre-trained VLM (e.g., Qwen2.5-VL-7B-Instruct) as an offline prompt extractor \(Y_{\mathrm{VLM}}\). For each low-resolution latent tile \(x_L^{(i)}\), a dedicated descriptive prompt is extracted via \(c_{\mathrm{local}}^{(i)} = Y_{\mathrm{VLM}}(x_L^{(i)}; \eta_i)\) prior to diffusion denoising. These compact prompts provide exact semantic tokens for local architectural details, legible signage, and surface textures, providing the necessary semantic constraints to eliminate hallucinations.

3. Dual Context Video Prompt Extraction: Resolving Spatio-Temporal Motion Ambiguities Extending localized prompting to video super-resolution introduces a new challenge: when presented solely with a small spatio-temporal crop \(x_L^{(i)}\), VLMs frequently suffer from a narrow temporal aperture and fail to recognize global object kinematics or scene transitions. To overcome this limitation, the framework introduces a dual-context prompt conditioning scheme: $$ c_{\mathrm{local}}^{(i)} := Y_{\mathrm{VLM}}(x_L^{(i)}, x_L; \eta_i) $$ By providing the full low-resolution sequence \(x_L\) alongside the target local volume \(x_L^{(i)}\) through structured system instructions, the VLM utilizes the full-video context to accurately disambiguate local motion dynamics (e.g., describing "rapidly rotating wheels" instead of a generic "vehicle"). This simultaneously resolves spatial misguidance and temporal misguidance.

A Worked Example

Consider an urban scene image of size 512×512 upscaled by 4× to 2048×2048: 1. Partitioning and Prompting: The image is partitioned into 25 latent tiles of size 64×64 with an overlap stride of 16. For a corner tile containing a storefront, a standard baseline broadcasts the global prompt "Aerial view of a city street with cars, pedestrians, and high-rise buildings", causing the store signage to dissolve into illegible geometric noise. In contrast, Tiled Prompts prompts the VLM on that specific tile to produce "Wooden coffee shop signboard with crisp engraved lettering reading COFFEE and golden trims". 2. Reverse Denoising: Over diffusion timesteps \(\tau_T \dots \tau_1\), DiT4SR evaluates local noise predictions \(\hat{e}^{(i)}\) conditioned on \(c_{\mathrm{local}}^{(i)}\) and the tile latent. 3. Gaussian Aggregation: Tile predictions are blended onto the canvas via Gaussian weighting window \(w_i\). The final 2048×2048 decoded image exhibits sharp, legible typography and authentic textures matching the prompt.

Key Experimental Results

Main Results

Quantitative evaluations were conducted across real-world image benchmarks (LSDIR1K, Urban100, OST300) upscaled 4× to 2048×2048 using DiT4SR, and real-world video benchmarks (VideoLQ, RealVSR, MVSR4x) using STAR (I2VGen-XL backbone).

Dataset	Prompt Strategy	NIQE↓	MUSIQ↑	MANIQA↑	CLIPIQA↑	CLIP Score↑	ImageReward↑	HPSv2↑
LSDIR1K	Null Prompt (No Text)	3.4537	62.2188	0.6126	0.6346	—	—	—
	Global Baseline	2.9427	63.8677	0.6373	0.6886	25.3348	-1.5901	0.1589
	Global + Local	2.9418	64.0749	0.6379	0.6932	25.7925	-1.4775	0.1719
	Local (Tiled Prompts, Ours)	2.9040	63.9731	0.6350	0.6917	27.2274	-0.6771	0.2011
URBAN100	Global Baseline	3.6156	53.1372	0.6668	0.6715	25.9807	-1.1688	0.1780
	Local (Tiled Prompts, Ours)	3.5001	54.9203	0.6618	0.6780	27.4044	-0.5193	0.2092
OST300	Global Baseline	2.9547	66.3813	0.6536	0.6799	25.1512	-1.6638	0.1418
	Local (Tiled Prompts, Ours)	2.9007	66.7233	0.6518	0.6943	26.8594	-0.7881	0.1838

In video super-resolution (VideoLQ benchmark), localized prompts achieved substantial improvements: NIQE dropped from 4.8016 to 4.5913, MUSIQ improved from 43.9160 to 45.7662, FAST-VQA increased from 0.7507 to 0.7643, DOVER rose from 53.5704 to 54.4158, and VQAScore surged from 0.3987 to 0.5542.

Ablation & Inference Runtime

Reference-based reconstruction metrics on high-resolution ground-truth sets (Table 3) alongside inference runtime benchmarks (Table 4) are reported below:

Dataset / Benchmark	Prompt Strategy	PSNR↑	SSIM↑	LPIPS↓	DISTS↓	FID↓	Inference Time (s)
LSDIR2048 (Ref)	Global Baseline	22.55	0.5973	0.3000	0.1462	17.09	—
	Global + Local	22.54	0.5980	0.2967	0.1445	17.98	—
	Local (Tiled Prompts, Ours)	22.63	0.6011	0.2900	0.1400	16.42	—
SEPE8K (Ref)	Global Baseline	23.00	0.6358	0.2852	0.1324	26.00	—
	Local (Tiled Prompts, Ours)	22.96	0.6335	0.2809	0.1276	24.16	—
DiT4SR Runtime	Baseline (Global Prompt)	—	—	—	—	—	162.58s
(25 tiles)	+ Tiled Prompts (Ours)	—	—	—	—	—	166.15s (+2.2%)
STAR Runtime	Baseline (Global Prompt)	—	—	—	—	—	1273.9s
(12 blocks)	+ Tiled Prompts (Ours)	—	—	—	—	—	1348.3s (+5.8%)

Key Findings

• Local prompts outperform concatenated Global+Local conditioning: Appending the global caption to the local prompt (Global + Local) performs inferior to using purely local prompts (Local), with ImageReward dropping from -0.6771 down to -1.4775 on LSDIR1K. This proves that restoration performance is driven by semantic specificity and the elimination of irrelevant concepts, rather than token quantity.
• Robustness under elevated CFG scales: Ablation across guidance scales \(s\) reveals that as CFG increases, global prompt baselines degrade rapidly due to linear amplification of the misguidance vector \(s\delta_i(c)\). In contrast, Tiled Prompts maintains stable and widening performance advantages under high guidance scales.
• Negligible computational overhead: Extracting 25 image prompts via VLM adds only ~3.57 seconds to a 162-second diffusion inference pipeline (+2.2%), while video extraction adds only +5.8%, establishing an exceptionally favorable efficiency-quality trade-off.

Highlights & Insights

• Theoretical rigor: Unifies diffusion and flow matching score discrepancies under an information-theoretic mutual information lower bound, rigorously validating the necessity of local conditioning.
• Dual-context video prompting: Solves the localized temporal blindness of VLMs by combining panoramic temporal references with localized window queries.
• Zero-training plug-and-play: Seamlessly integrates with any latent tiling diffusion backbone without altering pre-trained weights or requiring specialized fine-tuning.

Limitations & Future Work

• Dependence on VLM visual robustness: In scenarios with extreme sensor degradation, severe blur, or heavy compression noise, the VLM prompt extractor may hallucinate inaccurate semantics.
• OCR limits on microscopic text: General-purpose VLMs occasionally misidentify extremely low-resolution, degraded typographic symbols.
• Future directions: Exploring dynamic timestep prompt scheduling—applying localized semantic text guidance only during early structural denoising steps, followed by unconditional or high-frequency detail refinement in later steps.

Related Work & Insights

• vs SUPIR / SeeSR: While SUPIR and SeeSR employ VLMs or prompt extractors (DAPE) at the whole-image level, they revert to global broadcasting during latent tiling. Tiled Prompts operates at the patch level to bridge this local semantic gap.
• vs MultiDiffusion / Mixture of Diffusers: Prior latent tiling works concentrate on latent blending weights and boundary smoothness. Tiled Prompts addresses the orthogonal and fundamental problem of conditional semantic accuracy.
• vs Upscale-A-Video / STAR: Video diffusion SR models typically rely on static or short global prompts, ignoring localized motion variations; the proposed dual-context VLM prompting provides an effective paradigm for spatio-temporal video tiling.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates and mathematically bounds the prompt misguidance problem in latent tiling with exceptional clarity.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks spanning both high-resolution image and video super-resolution datasets with both reference and non-reference metrics.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical derivation accompanied by clear conceptual presentation and empirical validation.
• Value: ⭐⭐⭐⭐⭐ Provides a practical, plug-and-play solution for ultra-high-resolution generative image and video restoration.