WarpI2I: Image Warping for Image-to-Image Translation¶

Conference: ECCV2026
arXiv: 2606.31018
Code: https://shenzheng2000.github.io/WarpI2I.github.io/ (Project page, code to be confirmed)
Area: Image Generation / Image Translation
Keywords: Image Warping, Latent Diffusion Models, Image-to-Image Translation, Relighting, Saliency-guided

TL;DR¶

This paper proposes a model-agnostic, parameter-free warp-unwarp framework that spatially enlarges salient regions before encoding and subsequently reverses the warping, preserving fine-grained structural information in LDM image translation without increasing the latent space resolution. Meanwhile, a lightweight FLUX-based synthetic data pipeline is proposed, yielding significant improvements with minimal overhead for human/driving scene relighting and weather/time-of-day driving translation tasks.

Background & Motivation¶

Image-to-image (I2I) translation has made profound progress in tasks such as human relighting and weather translation in driving scenes, with Latent Diffusion Models (LDMs) being the mainstream approach. However, for high-resolution inputs, the encoders of lightweight LDMs compress the input with an 8x spatial downsampling. This causes fine-grained foreground structures, such as facial features, traffic signs, and license plates, to occupy extremely few pixels in the latent space, often resulting in blurriness and distortion in these areas within the generated results. An intuitive solution is to decrease the compression ratio or increase the input resolution to obtain a larger latent space, but this scales the computational cost quadratically with spatial size, making it extremely expensive for practical deployment. Although multi-scale diffusion or test-time refinement can alleviate this issue, they introduce additional training/inference overhead and architectural modifications.

The Key Challenge of this dilemma lies in: the computation-accuracy trade-off of LDMs demands a latent space as small as possible, but a small latent space inevitably discards high-frequency structural information. Existing methods either sacrifice efficiency for spatial resolution or permanently lose details during the encoding-decoding process—no work has investigated "whether the information can be redistributed before encoding". Ours' Key Insight is remarkably clever: since the problem stems from the encoder's uniform compression giving each position the same pixel budget, while semantically important regions (faces, eyes, small objects) actually occupy only a tiny fraction of the input, one only needs to spatially enlarge these regions locally before encoding, allowing them to occupy more pixels in the latent space, and then reverse the transformation afterwards. Core Idea: Apply a content-aware, saliency-guided warp operation to spatially enlarge (warp) important regions prior to LDM encoding. This concentrates the limited latent space pixels on semantically critical areas, and subsequently uses an inverse transformation (unwarp) to restore the original geometry after translation, thereby improving the fine-grained fidelity of any I2I framework with zero structural modifications.

Method¶

Overall Architecture¶

The core pipeline of WarpI2I is a three-stage "warp \(\to\) translation \(\to\) unwarp" process that can be plugged into any existing LDM image translation model. The overall architecture is as follows:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image I"] --> B["Saliency-Guided<br/>Spatial Warping (Warp)"]
    B --> C["Warped Image I'<br/>(Salient regions enlarged)"]
    C --> D["LDM Image Translation<br/>(pix2pix-Turbo /<br/>CycleGAN-Turbo)"]
    D --> E["Translated Image O'<br/>(Still in warped geometry)"]
    E --> F["Inverse Warping (Unwarp)"]
    F --> G["Final Output O<br/>(Restored geometry,<br/>with better details)"]
    B --> H["Saliency Map Construction<br/>(Detector + Kernel Density Estimation)"]
    H --> B

The input image I first passes through a saliency detector (e.g., InsightFace for faces and eyes, YOLO-World for objects) to obtain regional bounding boxes, which are then used to construct a continuous saliency map S via kernel density estimation. Based on S, a non-uniform warping (warp) is applied to I, which spatially enlarges high-saliency regions and compresses low-saliency regions, yielding the warped image I'. Subsequently, I' is fed into an existing LDM image translation model (such as pix2pix-Turbo or CycleGAN-Turbo), which outputs the translated result O' that is still in the warped geometry. Finally, an approximate inverse warping (unwarp) is executed on O' to restore the original geometric structure, resulting in the final output O. The entire warp and unwarp operations only introduce approximately 6 ms (3 ms each) of inference latency, with zero additional learnable parameters.

Key Designs¶

1. Saliency-Guided Spatial Warping: Redistributing Pixel Budgets Prior to Encoding

The uniform spatial downsampling of the LDM encoder implicitly assumes that all areas of an image carry equal informational importance. This is clearly invalid for semantic tasks—the face, especially the eye region, conveys significantly more discriminative information than the background. The Key Insight of this paper is that rather than modifying the encoder architecture or expanding the latent space, it is better to apply a non-uniform warping directly to the input image before encoding. This locally enlarges high-saliency regions in the pixel domain, which is equivalent to allowing them to occupy more latent pixels in the latent space.

Specifically, given a set of bounding boxes output by the detector (with centers c_i, widths w_i, and heights h_i), a continuous saliency map is constructed using Gaussian kernel density estimation:

\[S = \sum_{(c_i, w_i, h_i) \in \text{bboxes}} \mathcal{N}\left(c_i, b \begin{bmatrix} w_i & 0 \\ 0 & h_i \end{bmatrix} \right)\]

where b is the warping bandwidth controlling the deformation intensity (smaller b values yield more intense deformation; values between 64 and 256 demonstrate stable performance in experiments). The warping is parameterized using inverse mapping: \(I'(u) = I(T_S^{-1}(u))\), meaning that for each output pixel position u, the algorithm queries the saliency map S to locate its source position in the input space and then performs bilinear sampling. Since this warping is content-aware, salient regions (e.g., faces, eyes) are scaled up several times, preserving richer structural details during the subsequent encoding phase.

2. Pixel-Domain Inverse Warping Approximation: Differentiable and High Fidelity

Unlike previous warping-based approaches (such as LZU or InstanceWarp) that perform inverse transformations in the 2D feature space or prediction space, WarpI2I directly performs unwarping in the pixel domain (image space). This choice is motivated by three critical reasons: first, compatibility—ViT/DiT-like backbones operate on 1D token sequences, making them incompatible with 2D feature space unwarping; second, while the inverse transformation function \(T^{-1}\) lacks a closed-form solution, a piecewise bilinear approximation \(\tilde{T}^{-1}\) can efficiently construct a piece-wise locally invertible mapping; third, and most critical Key Insight: unwarping is differentiable, meaning the LDM can automatically learn to compensate for any minor artifacts introduced by the warping/unwarping process during training. Experiments also confirm that after the complete warp \(\to\) diffusion \(\to\) unwarp pipeline, the pixel difference heatmap with the original image is almost completely black (see the heatmap in Fig. 5), indicating minimal information loss.

3. FLUX-Based Lightweight Synthetic Data Pipeline: 10K Pairs are Sufficient

Real-world pixel-aligned data of the same scene under different illumination conditions is extremely difficult to acquire (as human poses, lighting, and backgrounds in dynamic scenes change simultaneously). Existing approaches (e.g., IC-Light, DreamLight) rely on complex, closed-source pipelines to generate millions of synthetic data samples, which is immensely time-consuming. This work proposes an exceptionally lightweight, open-source pipeline with the following core process:

FLUX Outpainting: Backgrounds are outpainted using FLUX.1-Fill-dev on an expanded canvas to alter the background while preserving the foreground subjects. ChatGPT is first used to generate structured descriptions of the original image (subject attributes + background + lighting). The background description is subsequently substituted with a neutral "indoor white studio" description (Background Prompt Substitution, BPS) to eliminate the ill-posed one-to-many mapping caused by training with the original white background.
Depth Anything Estimation: Depth maps are estimated for the outpainted images to provide geometric conditioning signals.
FLUX Depth-Conditioned Generation: The depth maps of the base image and the relighted image are stitched into a 2x1 grid. This is then fed into FLUX.1-Depth-dev with structured prompts to generate paired left-right images that retain foreground identity under different lighting conditions.
ChatGPT Automatic Filtering: GPT-4o is employed to automatically verify identity, clothing, pose consistency, and lighting fidelity of each image pair, achieving a pass rate of approximately 96% (with around 95% alignment between human reviews and ChatGPT decisions).

This pipeline requires only 10K synthetic pairs (taking about 1 day to generate on 8 A6000 GPUs), whereas IC-Light requires >10M and DreamLight requires >1M training images, achieving 2-3 orders of magnitude greater efficiency.

Loss & Training¶

For supervised paired tasks (human/driving scene relighting), the standard adversarial loss + L1 reconstruction loss + perceptual loss of pix2pix-Turbo are used. For unsupervised paired tasks (weather/time-of-day driving translation), the cycle-consistency loss + adversarial loss of CycleGAN-Turbo are employed, relying on the cycle-consistency of CycleGAN to automatically establish cross-domain mappings. WarpI2I itself does not introduce additional loss terms—warp and unwarp are inserted as pre-processing and post-processing modules without altering training objectives, making them easily compatible with LoRA fine-tuning.

Key Experimental Results¶

Main Results¶

Human Relighting User Study (VITON-HD): 60 participants rated four dimensions on a 1–5 scale.

Method	Subject Identity	Clothing Identity	Image Quality	Lighting Fidelity	Average
IC-Light	3.43	3.46	3.47	3.51	3.47
DreamLight	3.52	3.50	3.50	3.41	3.48
Ours (no BPS)	3.70	3.72	3.62	3.64	3.67
Ours (no warp)	3.60	3.62	3.48	3.54	3.56
Ours	4.36	4.43	4.31	4.21	4.33

Driving Translation Task (BDD100K Day \(\to\) Night): Quantitative comparison on automatic metrics like FID/KID.

Method	FID↓	KID↓×1000	Clean-FID↓	DINO-Struct↓×100
CycleGAN-Turbo	19.2	8.08	31.3	3.00
+Warp (Det bbox)	17.7	6.70	17.3	2.95
+Warp (GT bbox)	17.5	6.61	17.2	2.92

Ablation Study¶

Configuration	User Study Average Score	Description
Full model (Ours)	4.33	Full warp + BPS
w/o BPS	3.67	Without Background Prompt Substitution; significant drop
w/o Warp	3.56	Without warp operation; significant drop
Bw=64	4.27	Small bandwidth; handles stably
Bw=256	4.29	Large bandwidth; similarly stable
bbox jitter=0.2	4.24	Robust under detection errors
bbox jitter=0.3	4.15	Slight performance drop under large noise but still far superior to baseline

Key Findings¶

Warp Contributes the Most: Upgrading from w/o Warp (3.56) to Full (4.33) yields a 0.77 point user rating boost, greatly exceeding the contribution of BPS (3.56 \(\to\) 3.67, +0.11). This highlights warp itself as the primary source of improvement.
BPS Resolves Ambiguous Mapping under White Backgrounds: Backgrounds in VITON-HD are mostly pure white. Without BPS, the background diversity of generated results is constrained and training becomes unstable. Although BPS improves the quantitative score by only 0.11, the naturalness of the background in qualitative results improves significantly.
Highly Robust to Detection Errors: Even when adding Gaussian jitter with a standard deviation of 0.2–0.3 times the box width to the bounding boxes, the ArcFace identity similarity only drops slightly from 0.783 to 0.767–0.775, demonstrating that the warp mechanism does not require pixel-perfect segmentation.
Implicit Background Transformation: Although warping is designed to improve foreground structures, background regions also benefit as the spatial scope for diffusion model hallucination is confined (since the background is compressed after warping), making background color and lighting consistency more natural.

Highlights & Insights¶

"No Increase is the Best Increase": Achieving a jump in fine-grained fidelity from 3.5 to 4.3 solely through a fixed warp operation—without increasing the latent space, modifying the encoder architecture, or adding parameters—offers a highly inspiring and minimalist design philosophy.
Pixel-Domain Unwarping is a Masterstroke: Previous methods mostly executed unwarping in the feature or prediction domain, rendering them incompatible with ViT/DiT architectures. WarpI2I returns to pixel-domain inverse warping, which is not only compatible with arbitrary backbones but also leverages differentiability to let the diffusion model automatically compensate for minor artifacts from the warping kernel.
Overwhelmingly Efficient Data Pipeline: 10K vs. >1M training pairs combined with a 96% automatic filtering pass rate. The entire pipeline is open-source and reproducible, dramatically lowering the entry barrier for relighting research.
The Simple Cleverness of BPS: Automatically generated captions describe "white backgrounds" literally, which leads to an ambiguous mapping where inputs are always white but diverse outputs are desired. Simply replacing this with a neutral description elegantly addresses the issue, serving as an excellent example of a high-ROI heuristic.

Limitations & Future Work¶

Currently, validation is limited to single-step diffusion models (pix2pix-Turbo, CycleGAN-Turbo), without extensive tests on multi-step diffusion (e.g., full 50-step sampling with SDXL). The authors noted that in extreme weather-to-clear translations (such as Dense Fog \(\to\) Cityscapes), the baseline model itself diverges, which warping cannot fully resolve, indicating that a stronger baseline might be required for extreme tasks.
Frame-by-frame video inference may suffer from flickering during rapid movements. The authors suggested optical flow constraints as a potential direction for improvement but left it unexplored, representing a clear next step.
The background diversity of the synthetic data pipeline depends heavily on the outpainting quality of FLUX. Hard-coupled foreground and background properties (like reflective transparent objects) could cause inconsistencies during outpainting.

vs. InstanceWarp / Fovea / TPP: These methods leverage saliency warping to improve the performance of small targets in discriminative tasks (detection, segmentation). WarpI2I pioneering its application to generative image-to-image translation. The core difference lies in the choice of feature-domain vs. pixel-domain unwarping, which allows WarpI2I to remain compatible with arbitrary backbones.
vs. IC-Light / DreamLight: While they deliver respectable relighting performance, they demand massive synthetic datasets (>1M-10M pairs). WarpI2I's data pipeline is more efficient by orders of magnitude, and its warping mechanism yields significantly superior identity and clothing fidelity.
vs. Other Pixel Allocation Strategies: Although multi-scale diffusion (Cascaded Diffusion) and test-time refinement preserve details, they introduce significant computational overhead, rendering them impractical for deployment. WarpI2I's warping comes with virtually zero cost, and the concept of "allocation" itself could be extended to other vision tasks requiring prioritized attention.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Introducing warping operations into LDM image translation is elegantly simple and counter-intuitive ("tweaking the inputs instead of modifying the models"), while the synthetic data pipeline significantly lowers the threshold for relighting research.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across three tasks (human, driving scene, and weather translation) x three baselines x four datasets x two ablation dimensions, featuring a 60-person user study, automatic metrics, and robustness tests against detection errors and warping bandwidths.
Writing Quality: ⭐⭐⭐⭐ Clear motivation, high-quality figures, and rigorous arguments. Although section 3 includes several sub-sections that somewhat embed the core warping mechanism deep in data pipeline details, the overall paper remains highly readable.
Value: ⭐⭐⭐⭐⭐ Extremely simple yet highly effective. Its model-agnostic nature allows it to be plug-and-play in any I2I task. The open-source, reproducible data pipeline holds outstanding practical value.