Learn Once, Edit Anywhere: Visual Direction Transfer for Diffusion Models¶

Conference: ECCV 2026
arXiv: 2403.19645
Project Page: vidit-edit.github.io
Code: None
Area: Diffusion Models / Image Editing
Keywords: Diffusion Models, Image Editing, Visual Direction Transfer, Disentangled Editing, Zero-shot Editing

TL;DR¶

ViDiT proposes a "learn once, edit anywhere" framework: it optimizes a continuous latent direction d from a small number of image edit pairs (before/after) and injects this direction into the CFG inference process of a diffusion model. This achieves zero-shot, disentangled, and intensity-controllable fine-grained attribute editing on arbitrary images, without fine-tuning the base model or requiring image-specific optimization.

Background & Motivation¶

Diffusion models have achieved great success in text-to-image generation, and existing image editing methods almost entirely rely on natural language descriptions to specify the editing intention (e.g., SEGA, InstructPix2Pix, Prompt2Prompt). However, natural language has a fundamental "description bottleneck": many fine-grained visual changes—precise facial structural adjustments, subtle texture transfers, and stylized brushstroke changes—are obvious at a glance between two images but are extremely difficult to describe precisely in words. A prompt like "add a beard to this person" is not only a coarse prior but also inevitably entangles the edit with other attributes (e.g., age, skin tone). Meanwhile, the latent space of GANs has been proven to possess rich, linearly manipulable, and disentangled semantic directions (e.g., StyleSpace), but the recursive denoising architecture of diffusion models and the complex variable management across timesteps make finding similar directions extremely difficult, with only a few disentangled directions discovered so far. The Key Challenge is that the expressiveness of visual differences far exceeds text, yet existing diffusion editing tools are restricted by text interfaces. The Core Idea of this work is to directly transfer the self-evident visual delta (\(\Delta x = x' - x\)) in image edit pairs into a compact, continuous editing direction d in the conditioning space of the diffusion model, thereby bypassing the text bottleneck and achieving "learn once, edit anywhere."

Method¶

Overall Architecture¶

The goal of ViDiT is to learn a continuous latent direction d from a set of image edit pairs {(x, x')} such that it can replicate the semantic transformation from x to x'. The entire pipeline is divided into two phases: (1) Direction Learning Phase—given N pairs of pre- and post-edited images, a direction d is randomly initialized in the text embedding space of Stable Diffusion and iteratively optimized via a dual-objective loss function (semantic alignment + latent alignment), during which the denoising network ε_θ and the CLIP image encoder E_I are frozen; (2) Inference Editing Phase—the optimized d is injected into the CFG formula, replacing the original text conditioning residual term, to exert zero-shot editing on any input image (including real photos inverted via DDPM) with the edit intensity controlled by λ_e. Multiple independently learned directions can be additively combined during inference to achieve simultaneous multi-attribute editing.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image Edit Pairs<br/>(x, x') from arbitrary supervision sources"] --> B["Randomly initialize direction d<br/>in text embedding space"]
    B --> C["Semantic Alignment Loss<br/>Frozen CLIP encoder<br/>Contrastively pull d close to x' and push away from x"]
    B --> D["Latent Alignment Loss<br/>Frozen denoising network ε_θ<br/>Maximize difference in noise prediction before & after editing"]
    C --> E["Optimized Direction d*"]
    D --> E
    E --> F["Inference: Inject direction into CFG<br/>λ_e controls edit intensity<br/>Zero-shot edit arbitrary images"]
    F --> G["Additive Combination of Multiple Directions<br/>Simultaneously edit multiple attributes"]

Key Designs¶

1. Semantic Alignment Loss: Anchoring d to the Correct Semantic Concept with CLIP Contrastive Learning

The primary challenge in optimizing a standalone direction vector d is: how to make d "know" what semantic transformation it should represent? ViDiT utilizes a frozen CLIP image encoder E_I as a global semantic anchor. The core intuition is that if d truly represents the editing direction from x to x', then in the CLIP feature space, d should be highly similar to the edited image x' and dissimilar to the original image x. Thus, a contrastive semantic alignment loss is constructed: \(\mathcal{L}_{\text{sem}} = 1 - \text{cossim}(E_I(x'), \mathbf{d}) + \text{cossim}(E_I(x), \mathbf{d})\). This loss pushes d into the "semantic neighborhood" of the target concept, ensuring the direction is semantically correct at a high level. However, CLIP's globally pooled features lack spatial resolution—a direction learned using only L_sem might be semantically correct (e.g., indeed adding a beard) but will lead to changes in unrelated attributes like face shape or skin tone because CLIP cannot distinguish between "beard texture changes" and "jawline shape changes."

2. Latent Alignment Loss: Locking Spatial Details and Identity Consistency with the Denoising Network

To address the spatial granularity limitation of L_sem, ViDiT introduces a second loss, L_latent, which operates directly inside the denoising network of the diffusion model. The key observation is that unlike GANs which condition on compact, low-dimensional latent codes, the denoising network of a diffusion model outputs spatially aligned, high-dimensional noise prediction maps \(\epsilon_\theta(x_t, \mathbf{d})\), which encode dense pixel-wise information. L_latent maximizes the difference in noise predictions for the pre- and post-edited images conditioned on the same direction d: \(\mathcal{L}_{\text{latent}} = -\mathbb{E}_{x_0,\epsilon,t}[||\epsilon_\theta(x'_t, \mathbf{d}) - \epsilon_\theta(x_t, \mathbf{d})||_2^2]\). Intuitively, this forces the noise predictions generated by d to maximize the distinction between the pre- and post-edited images, thereby "grounding" the direction into the model's native denoising representation space. Experiments demonstrate that L_latent is crucial for preserving high-frequency identity cues such as hair strands, eyeglass reflections, and skin texture. Without it, the edits suffer from severe attribute entanglement (e.g., adding "grey hair" inadvertently alters skin tone).

3. Edit Anywhere Inference & Multi-Direction Combination: Modifying CFG for Zero-shot, Intensity-Controllable, and Multi-Attribute Editing

Once d is learned (each concept only needs to be learned once), it acts like a new "token" in the diffusion model's vocabulary. During inference, ViDiT modifies the standard CFG formula, replacing the original text-dependent residual term with a visual delta residual: \(\bar{\epsilon}_\theta(x_t, c, \mathbf{d}) = \tilde{\epsilon}_\theta(x_t, c) + \lambda_e(\epsilon_\theta(x_t, \mathbf{d}) - \epsilon_\theta(x_t, \phi))\). Here, \(\lambda_e \in \mathbb{R}\) provides continuous and fine-grained control over the edit intensity: setting it to 0 restores the original image, positive values enhance the editing effect, and negative values achieve reverse editing. Since editing directions are injected additively, multiple independently learned directions can be directly combined during inference—\(\hat{\epsilon}_\theta = \tilde{\epsilon}_\theta + \sum_i \lambda_{e_i}(\epsilon_\theta(x_t, d_i) - \epsilon_\theta(x_t, \phi))\)—without any retraining. For example, one can simultaneously apply "smile" + "Asian" + "beard" directions to a face, with each direction remaining highly disentangled. For real photos, DDPM Inversion is first used to invert the input into the model's latent trajectory, and the direction is then injected during the reverse denoising process. The entire inference takes about 5 seconds (on a single NVIDIA L40 GPU).

A Complete Example: Learning the "Beard" Editing Direction¶

To illustrate the complete workflow, consider learning the "beard" editing direction. The supervision source uses StyleGAN2 trained on FFHQ to generate 1,000 paired "no beard/beard" images {(x_i, x_i')}. d is randomly initialized in the text embedding dimension of Stable Diffusion. At each iteration: a pair of images is randomly sampled, and the same noise and timestep t are applied to both to obtain x_t and x'_t. L_sem is computed by passing x and x' into the frozen CLIP image encoder and performing cosine similarity contrast with d. L_latent is computed by feeding x_t and x'_t into the frozen denoising network (conditioned on d) and maximizing the L2 distance between their noise predictions. The total loss L = L_sem + L_latent is backpropagated to update only d. After 1,000 steps of AdamW optimization, d successfully captures the semantic transformation of "adding a beard." During inference, a new real face photo is inverted via DDPM to obtain its noise sequence. In each step of reverse denoising, d is injected into the CFG formula with λ_e = 1.0, producing the edited image with a beard while preserving identity, background, and lighting perfectly.

Loss / Training¶

The total training objective is a direct sum of the two losses: \(\mathcal{L} = \mathcal{L}_{\text{sem}} + \mathcal{L}_{\text{latent}}\), without any balancing weights. The optimizer is AdamW with a learning rate η, training for 1,000 epochs, with each epoch traversing the entire set of N=1000 samples. Throughout the training process, the denoising network ε_θ of Stable Diffusion and the CLIP image encoder E_I are completely frozen, and only the direction vector d is updated. d is initialized as a random vector in the text embedding space. Training takes only a few minutes to complete on a single GPU.

Key Experimental Results¶

Main Results¶

On 200 pairs of input-edited images (covering two semantics: "Asian" and "smile"), ViDiT is compared against four text-driven editing methods. ViDiT does not use any target text descriptions and relies solely on the directions learned from visual pairs. The results show that ViDiT comprehensively leads in identity preservation metrics, with LPIPS as low as 0.030 (compared to 0.059 for InstructPix2Pix) and DINO as high as 0.929, while semantic alignment metrics remain competitive with text-reliant methods.

Method	LPIPS↓	CLIP-T↑	DINO↑	SigLIP-T↑	DreamSim↑
SEGA	0.179	0.388	0.714	0.134	0.757
Prompt2Prompt	0.074	0.408	0.867	0.143	0.869
InstructPix2Pix	0.059	0.403	0.851	0.145	0.877
Concept Sliders	0.121	0.325	0.842	0.097	0.844
ViDiT (Ours)	0.030	0.407	0.929	0.139	0.905

Comparisons with visually-conditioned methods (on 100 generated samples) also validate the superiority of ViDiT: achieving 0.831 for CLIP-I, 0.913 for SigLIP-I, and 0.886 for DINO, which are significantly better than methods requiring extra training or adapter modules, such as Weights2Weights (DINO 0.642), IP-Adapter (DINO 0.767), and W+ Adapter (DINO 0.759). In the user study (40 participants, 60 input-edited image pairs, rated on a 1-5 scale), ViDiT obtained the highest average preference score of 3.36 (compared to 3.25 for Concept Sliders and 2.74 for Prompt2Prompt).

Ablation Study¶

Setup	LPIPS↓	CLIP-T↑	DINO↑	SigLIP-T↑	DreamSim↑	Notes
N=10	0.098	0.403	0.869	0.143	0.872	Captures meaningful semantics with only 10 pairs
N=100	0.136	0.425	0.842	0.148	0.771	CLIP-T peaks at N=100
w/o L_latent	0.121	0.409	0.832	0.143	0.860	Severe editing entanglement without latent alignment
Ours (ViDiT, N=1000)	0.093	0.406	0.891	0.147	0.861	Full model achieves the best identity preservation

Additionally, ablation over inference timesteps shows that injecting the direction throughout the full T timesteps yields the best balance between edit fidelity and identity preservation; reducing this to 0.4T produces a stronger editing effect but causes noticeable identity shifts. DDPM Inversion significantly outperforms DDIM Inversion in preserving high-frequency details (hair, frames, skin tone). Rescoring analysis validates the disentanglement of the edits: applying four editing directions (Asian/smile/gender/beard) to 100 images shows significant changes in CLIP classification probability along the diagonal (+53.6% for Asian, +41.2% for smile, +94.7% for gender, +28.3% for beard), while off-diagonal interactions are generally minor, demonstrating that each direction is independently controllable.

Key Findings¶

L_latent is a critical component: Removing L_latent drops DINO from 0.891 to 0.832, and qualitative results show obvious attribute entanglement (e.g., adding "grey hair" alters skin tone), demonstrating that the global semantics of CLIP alone are insufficient for maintaining fine-grained identity consistency.
Extremely high sample efficiency: A meaningful editing direction can be learned with as few as N=10 pairs (DINO 0.869), although a larger N=1000 yields stronger generalization (DINO 0.891) as more samples cover a broader variation of the target concept.
CLIP-T peaks at N=100 (0.425), suggesting a mild trade-off between semantic alignment and structural fidelity—semantic alignment drops slightly with too many samples, while structural fidelity continues to improve.
Continuous and controllable edit intensity: λ_e can be continuously adjusted from negative to positive values, enabling bidirectional editing in the same direction (e.g., "adding beard" / "removing beard") without extra training.

Highlights & Insights¶

The "visual delta transfer" paradigm: Reframing the image editing problem from "writing prompts" to "transferring visual differences" bypasses the description bottleneck of natural language. This idea possesses broad transfer value—any editing source capable of producing before/after image pairs (GANs, domain adaptation models, or even Photoshop actions) can be plugged into the ViDiT framework.
Exquisite complementary design of dual losses: L_sem provides the high-level semantic direction ("where to go"), while L_latent provides low-level spatial constraints ("where to stop"). Summing them yields the optimal path. This two-tier optimization scheme of "semantic anchoring + spatial refinement" can be generalized to other conditional generative tasks.
Practical philosophy of leaving the base model untouched: The entire training process only optimizes a single vector d, while the denoising network and visual encoder are completely frozen. This means ViDiT serves as a plug-and-play addon for any pre-trained diffusion model, introducing zero risk of model drift or additional parameter overhead during inference.
Natural emergence of additive combinations: Since each direction is optimized independently and injected additively, multi-direction composite editing is directly available without any special design. This "composability" stems from the core abstraction of defining edits as direction vectors in the CFG residual space, rather than post-hoc engineering.

Limitations & Future Work¶

Disentanglement upper bound limited by the supervision source: The degree of disentanglement of the direction learned by ViDiT cannot exceed the disentanglement quality of the supervision image pairs themselves. If the editing direction of StyleGAN is inherently entangled (e.g., "age" is correlated with "grey hair/glasses"), ViDiT will inherit these biases.
Spatial limitations of a single global direction: Currently, ViDiT optimizes a single global direction d for each concept, which has limited expressive power for spatially non-uniform edits (e.g., editing only the left eye without affecting the right, or localized texture replacement) because a global vector cannot carry spatially varying editing demands.
Inherent biases of CLIP and Stable Diffusion: Rescoring analysis reveals some statistically expected cross-attribute shifts (e.g., "feminization" edits significantly decrease the "beard" score), which are inherited from the pre-trained components rather than introduced by ViDiT itself.
Scenarios with insufficient fine-grained signals: When the changes encoded by the supervision image pairs are too subtle (e.g., minute adjustments to nose shape), d may fail to learn enough semantic signal to faithfully replicate the edit, which is an inherent limitation of the information capacity of the diffusion model's conditioning space.
Future Work: Extending d to a spatially-varying direction map (an independent direction vector at each pixel location) is expected to support localized editing; introducing stronger editing supervision sources (such as more disentangled GAN direction discovery methods) can directly improve ViDiT's editing quality; extending the framework to object-level and scene-level editing is a natural and valuable direction.

vs Concept Sliders: Concept Sliders trains a LoRA adapter for each concept, which relies on text descriptions and requires fine-tuning model parameters. ViDiT does not train any model weights and only optimizes a single direction vector, which is more lightweight and free from text bottlenecks. However, the LoRA approach may have stronger expressive power for extremely fine-grained edits.
vs W+ Adapter: W+ Adapter trains additional adapter modules to map the W+ space of StyleGAN to the diffusion model. ViDiT directly learns directions in the conditioning space without introducing any new modules, which is simpler but potentially less flexible than dedicated mapping modules.
vs Prompt2Prompt / SEGA / InstructPix2Pix: These methods rely on text prompts to describe editing intentions, which are limited by language bottlenecks and prone to entanglement in fine-grained facial attribute editing. ViDiT learns directly from visual pairs, naturally avoiding text ambiguity, and outperforms all text-driven methods in identity preservation.
vs NoiseCLR: NoiseCLR discovers disentangled directions using contrastive learning inside the diffusion model, but is restricted to directions already learned within the model's representations. ViDiT can transfer semantics from arbitrary external sources (GANs, domain adaptation models, other diffusion editors), providing much more flexible and diverse direction sources.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Transferring the idea of GAN direction discovery to diffusion models and using visual pairs instead of text prompts for editing—the core idea is simple yet effective. While not ground-breaking, the framework offers a fresh perspective.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Ablations cover sample size, loss terms, inference timesteps, and inversion methods. It provides comprehensive comparisons with both text-driven and visually-conditioned methods. Additionally, user studies and rescoring analysis supplement perceptual and quantitative disentanglement evidence.
Writing Quality: ⭐⭐⭐⭐☆ Clear motivation (text description bottleneck → visual delta transfer), logically sound methodology, and rich figures and tables. The appendix provides sufficient supplementary experiments and failure case analyses.
Value: ⭐⭐⭐⭐☆ Source-agnostic framework, plug-and-play, training-free for the base model, and extremely fast inference, showcasing strong practicality. The "visual delta transfer" paradigm opens up a new space of ideas for diffusion model editing and could potentially inspire future work to bring semantic capabilities from more structured generative models into diffusion models.