Skip to content

LCM-Lookahead for Encoder-Based Text-to-Image Personalization

Conference: ECCV 2024
arXiv: 2404.03620
Code: https://lcm-lookahead.github.io/
Area: Image Generation
Keywords: Text-to-Image Personalization, Face Generation, LCM, Encoder-based, Identity Preservation

TL;DR

This paper proposes utilizing Latent Consistency Model (LCM) as a "shortcut" to enable backpropagation of image-space losses (e.g., identity recognition loss) during the training of diffusion model encoders. Combined with self-attention feature sharing and consistent data generation, this approach significantly enhances identity preservation and prompt alignment in encoder-based facial personalization.

Background & Motivation

  1. Background: Text-to-image personalization aims to enable pre-trained models to generate new images of specific user concepts, particularly faces. Optimization-based methods (such as DreamBooth and Textual Inversion) yield high-quality results but are slow, whereas encoder-based methods (such as IP-Adapter) are fast but offer poor identity preservation.

  2. Limitations of Prior Work: (1) Encoder-based methods are trained purely on diffusion denoising loss (\(L_2\) noise prediction), which prevents them from directly optimizing facial similarity using perceptual losses like identity losses as in GAN inversion; (2) diffusion training operates on intermediate timesteps, and the generated noisy/blurry approximated images cannot be effectively fed into identity recognition networks; (3) current encoders struggle to strike a balance between identity preservation and prompt editability.

  3. Key Challenge: The multi-step training mechanism of diffusion models makes it difficult to directly apply image-space losses. The standard single-step DDIM approximation (\(\hat{x}_0\)) is blurry and distorted at early timesteps, making it unsuitable for downstream perceptual networks, yet perceptual loss is crucial for enhancing identity preservation (as extensively validated in the GAN field).

  4. Goal: How can image-space identity loss be effectively integrated into the training of encoder-based diffusion personalization models?

  5. Key Insight: LCM (distilled from the same base model) can generate high-quality preview images from intermediate noisy latents in a single step while maintaining semantic alignment with the original model. Utilizing this "shortcut" allows for clean image previews to be obtained during training to compute perceptual losses.

  6. Core Idea: Leverage LCM-LoRA for single-step denoising to obtain high-quality preview images, compute identity loss through these previews to backpropagate gradients to the encoder, all while preserving the alignment between LCM and the base model.

Method

Overall Architecture

Based on the IP-Adapter Face model (with an SDXL backbone), three enhancements are proposed: (1) LCM-Lookahead Loss—computes identity loss using LCM single-step previews; (2) KV Encoder—extracts self-attention K/V features from the reference image and injects them into the denoising process; (3) Consistent Data Generation—generates multi-style training data of the same identity by exploiting the mode collapse property of SDXL-Turbo.

Key Designs

  1. LCM-Lookahead Loss:

    • Function: Obtains clean image previews via the LCM shortcut during training to calculate image-space identity loss.
    • Mechanism: Given a noisy latent \(z_{r,t}\), a single-step denoising with LCM-LoRA is performed to obtain the preview \(\hat{z}_{r,0}\). After decoding it into an image, the identity distance between the preview and the reference image is computed: \(\mathcal{L}_{LH} = \mathcal{D}(D_{VAE}(\hat{z}_{r,0}), I_c)\). The training concurrently utilizes the standard diffusion loss (via the base SDXL model) and the lookahead identity loss (via the LCM path).
    • Design Motivation: It yields a significantly higher approximation quality than standard \(\hat{x}_0\), producing clear previews even at early timesteps. The alignment between LCM and the base model ensures the semantic consistency of the preview.

  2. Alignment Preservation Strategy:

    • Function: Prevents the lookahead loss from disrupting the distribution alignment between LCM and the base model.
    • Mechanism: In half of the training iterations, the LCM-LoRA weights are randomly scaled \(\alpha_{LoRA} \in [0.1, 1.0]\) so that the encoder cannot learn solutions that only work for LCM but fail for the base model. Additionally, importance weighting (timestep annealing) is applied to bias sampling towards early noisy timesteps.
    • Design Motivation: Prolonged training using a fixed LCM path causes the encoder to overfit to LCM-specific behavior patterns, breaking the alignment with the base model. Random scaling acts as a regularization mechanism to find a more general solution.

  3. Self-Attention Feature Sharing (KV Encoder):

    • Function: Extracts visual appearance features from the reference image and injects them into the generation process.
    • Mechanism: The SDXL U-Net is replicated as a KV Encoder. The noisy latent of the reference image is passed through it, caching the self-attention K/V at each layer. During generation, these K/Vs are concatenated with the self-attention of the primary denoising UNet: \(K^l = K_{z_{r,t}}^l \odot K_{z_{c,t}}^l\).
    • Design Motivation: Inspired by video models and appearance transfer work, expanding self-attention allows the generated image to "see" the reference image's appearance features, boosting identity preservation. LoRA is used to fine-tune the KV Encoder to learn to discard style information irrelevant to identity.

  4. Consistent Data Generation:

    • Function: Generates multi-style training data for the same identity using the mode collapse of SDXL-Turbo.
    • Mechanism: The adversarial training of SDXL-Turbo leads to mode collapse; for a sufficiently detailed prompt describing a person, different seeds generate the same identity. This property is exploited to generate 500K images across 100K identities, with each identity rendered in various styles (e.g., oil painting, comic, pencil sketch).
    • Design Motivation: (1) Avoids privacy and ethical issues associated with collecting real face datasets; (2) the generated data includes stylized images, encouraging the encoder during training to decouple style and identity, thereby improving prompt alignment.

Loss & Training

\(\mathcal{L} = \mathcal{L}_{Diffusion} + \lambda \mathcal{L}_{LH}\). The base model branch employs the standard diffusion loss, while the LCM branch utilizes the identity loss (ArcFace). TinyVAE is used for decoding to save VRAM and improve gradient flow. The model is trained for 5,000 iterations with a batch size of 8 on 2x A100 GPUs.

Key Experimental Results

Main Results

Method ID Similarity↑ (FFHQ) CLIP Score↑ (FFHQ)
IP-Adapter (α=0.5) 0.28 0.285
IP-Adapter (α=1.0) 0.38 0.265
PhotoMaker 0.32 0.278
InstantID 0.42 0.280
Ours (LCM-Lookahead) 0.36 0.290

Ablation Study

Configuration ID Sim↑ CLIP↑ Description
Backbone (IP-A α=0.5) 0.28 0.285 Baseline backbone
+ LCM Loss 0.33 0.282 Obvious ID improvement
+ KV Encoder 0.35 0.286 Appearance transfer further improves results
+ Consistent Data 0.34 0.290 Prompt alignment significantly improved
Full 0.36 0.290 Full model

Key Findings

  • LCM previews are far superior to standard \(\hat{x}_0\) approximations: Visualizations show that LCM generates clear facial images even at early timesteps, whereas \(\hat{x}_0\) is blurry and chromatized.
  • Alignment preservation is crucial: Without maintaining alignment, the identity quality improves in the short term but collapses over extended training.
  • Consistent data contributes the most to editability: Training data containing stylized targets teaches the encoder to follow style edits while keeping the identity constant.
  • User study validation: In 460 responses, users showed a clear preference for this method over the backbone IP-Adapter.

Highlights & Insights

  • LCM as a training shortcut: This is a general technique—any scenario requiring image-space loss in diffusion training can use the LCM shortcut. It is not limited to identity loss; LPIPS/CLIP losses are also applicable.
  • Turning SDXL-Turbo's mode collapse into a feature: A typical flaw of generative models (mode collapse) is cleverly leveraged to generate consistent identity data, which is a highly creative approach.
  • Practical experience in alignment preservation: The combination of randomly scaling LoRA weights and timestep annealing offers a practical solution to maintain the alignment of distilled models.

Limitations & Future Work

  • InstantID still achieves higher identity similarity (but it was trained on 60M data using 48 GPUs).
  • LCM-Lookahead increases VRAM and computational overhead during training (due to the additional UNet forward pass).
  • The method has only been validated in the facial domain and needs expansion to general objects.
  • This method can be applied to stronger backbones such as InstantID to yield further improvements.
  • vs IP-Adapter: Serves as a direct improvement on IP-Adapter, elevating its capabilities by adding identity loss and a KV encoder.
  • vs InstantID: InstantID uses ControlNet to preserve pose along with larger-scale training; its identity preservation is stronger, but it severely limits pose diversity.
  • vs PhotoMaker: PhotoMaker utilizes dedicated ID datasets to modulate CLIP features, whereas LCM-Lookahead uses a more general loss mechanism.
  • vs PortraitBooth (Concurrent): PortraitBooth applies identity loss only at low-noise timesteps, which limits its influence on early stages. LCM-Lookahead is effective across all timesteps through high-quality previews.

Rating

  • Novelty: ⭐⭐⭐⭐ Innovative combination of the LCM shortcut and mode collapse exploitation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Quantitative + qualitative + user studies + comprehensive ablation.
  • Writing Quality: ⭐⭐⭐⭐ Clear logic with sufficient technical details.
  • Value: ⭐⭐⭐⭐ The LCM shortcut is a generic technique, extending its value beyond a single task.