Skip to content

ID-PreFeR: ID-Preserving Face Restoration with Mixed Data Quality

Conference: ECCV 2026
Paper: ECCV Official
Code: https://eccv.ecva.net/virtual/2026/poster/3808
Area: Image Restoration
Keywords: Face Restoration, Diffusion Model, Identity Preservation, Mixed Data Quality, Personalized Injector

TL;DR

Addressing the bottlenecks of heavy personalization burdens and severe degradation under mixed-quality references in personalized face restoration, ID-PreFeR introduces a regularization-free lightweight cross-attention LoRA injector, an identity-quality disentanglement strategy using MLLM quality tokens with Min-SNR-\(\gamma\) weighting, and an inference-time ID-preserving sampling scheme, achieving state-of-the-art identity fidelity and perceptual restoration even with only a single high-quality reference image.

Background & Motivation

Low-quality (LQ) facial images are pervasive in real-world scenarios, such as tiny faces in distant mobile captures, surveillance snapshots, or online portraits degraded by compression, noise, motion blur, and low resolution. Blind face restoration aims to reconstruct high-quality (HQ) counterparts from these degraded observations. However, face restoration is an inherently ill-posed inverse problem: a single heavily degraded LQ observation admits numerous perceptually plausible HQ solutions. Because human visual perception is acutely sensitive to subtle facial geometry and fine-grained landmark cues, minor structural deviations immediately result in perceptible identity drift. Prior reference-free methods leveraging GANs, VQ codebooks, or general diffusion priors lack subject-specific cues, frequently hallucinating generic features that fail to preserve the true identity under severe corruption.

To prevent identity drift, recent personalized face restoration methods introduce reference images of the target individual to guide the generative process. Nevertheless, existing paradigms confront two severe dilemmas. First, personalization imposes a heavy computational and storage burden. State-of-the-art personalized diffusion frameworks typically fine-tune most or all backbone parameters and persist a complete model per subject; furthermore, to suppress catastrophic forgetting and language drift, they require voluminous class-regularization datasets far larger than the reference set. Second, existing methods exhibit a fragile dependency on reference quality. They are typically learned under the strict assumption that all references are pristine HQ images. In practical mobile photo libraries, however, collected references inevitably contain blurry, noisy, or low-resolution shots. When the proportion of LQ references increases, existing models mistakenly encode the degradation patterns as intrinsic identity features, causing dramatic performance collapse.

Faced with the joint constraints of practical mobile deployment and imperfect, mixed-quality reference collections, this paper tackles the problem by disentangling identity from image degradation while constraining personalization to an ultra-compact parameter footprint. Core idea: construct a lightweight cross-attention LoRA injector adding only 0.25% parameters to the frozen diffusion backbone, eliminate external regularization data via a latent prior loss, isolate degradation artifacts through MLLM quality-aware tokens paired with Min-SNR-\(\gamma\) weighting, and enforce post-hoc identity fidelity via inference-time gradient-guided sampling.

Method

Overall Architecture

ID-PreFeR decouples the workflow into an efficient per-subject personalization phase (training) and a frozen-backbone conditional restoration phase (inference). During training, given a compact set of mixed-quality references of a target identity, the framework fine-tunes only the LoRA weights on the cross-attention layers of the SDXL U-Net decoder alongside learnable prompt tokens, while freezing the entire generative backbone. A latent prior loss computed against class prompts anchors the personalized branch to the pretrained backbone, completely eliminating the need for external regularization datasets. During inference, the learned 6.41M adapter is loaded on demand, conditional latent diffusion proceeds using ControlNet features extracted from the LQ input, and an ID-preserving sampling mechanism applies gradient ascent on intermediate noise estimates via a frozen face recognition network to maximize identity consistency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Small Mixed-Quality Reference Set"] --> B["Personalized Injector<br/>LoRA on Cross-Attention + Latent Prior Loss"]
    B --> C["ID-Quality Disentanglement<br/>MLLM Quality Tokens + Min-SNR-ฮณ Weighting"]
    C --> D["Subject-Specific LoRA Adapter (Only 6.41M Params)"]
    D --> E["LQ-Conditioned Reverse Diffusion<br/>Guided by Frozen SDXL Backbone & ControlNet"]
    E --> F["ID-Preserving Sampling<br/>Late-Stage ArcFace Gradient Ascet on Noise"]
    F --> G["Output: Identity-Preserved Restored Face"]

Key Designs

1. Lightweight Personalized Injector: Regularization-Free Adaptation via Latent Anchoring

To eliminate the massive computational overhead of full-model tuning and the need to store gigabytes of weights per identity, ID-PreFeR keeps the SDXL backbone \(\theta^*\) frozen and inserts low-rank adaptation (LoRA, rank \(r=16\)) exclusively into the cross-attention layers of the U-Net decoder. This restricts trainable parameters to 6.41M (only ~0.25% of SDXL, requiring ~13 MB storage in FP16). To prevent language drift without collecting thousands of auxiliary regularization images, the framework enforces a latent prior loss directly against the frozen backbone:

\[\mathcal{L}_{\text{reg}} = \|\epsilon_\theta(z_t, t, \tau(y^*)) - \epsilon_{\theta^*}(z_t, t, \tau(y^*))\|_2^2\]

where \(y\) contains the personalized identifier token (e.g., a photo of a [V] face) while \(y^*\) is the generic class prompt without the identifier (e.g., a photo of a face). By compelling the personalized denoiser to match the frozen backbone's noise predictions on generic class prompts in the latent domain, the model selectively updates identity-specific semantics when prompted with \([V]\) while preserving broad facial priors and general language understanding.

2. ID-Quality Disentanglement: Isolating Degradation Artifacts via Dual Mechanisms

When references contain blur, noise, or compression artifacts, naive personalization causes the model to fit these degradations as facial traits. ID-PreFeR addresses this through a two-fold disentanglement strategy. First, at the textual conditioning level, a lightweight MLLM classifier assigns binary quality labels (HQ or LQ) to all references, inserting learnable quality tokens \([Q] \in \{[Q_{\text{HQ}}], [Q_{\text{LQ}}]\}\) into the training prompt a [Q] [V] face. Initialized with the text embeddings of sharp and blurry, quality tokens are jointly optimized with the identity token \([V]\). At inference, quality tokens are omitted and only \([V]\) is passed, ensuring that \([Q]\) absorbs reference-specific degradation while \([V]\) encapsulates invariant identity attributes.

Second, considering the forward latent diffusion process where an LQ latent decomposes into \(z_{\text{LQ}} = z_{\text{HQ}} + \epsilon_{\text{deg}}\), the degraded latent yields an effective noise combination \(\epsilon + \omega_t \epsilon_{\text{deg}}\) where \(\omega_t = \sqrt{(1 - \bar{\alpha}_t)/\bar{\alpha}_t}\). At high timesteps (low SNR), \(\omega_t \to 0\) and degradation naturally vanishes; conversely, at low timesteps where the SNR is high, \(\omega_t \epsilon_{\text{deg}}\) dominates and forces the injector to reproduce degradation artifacts. To eliminate this issue, ID-PreFeR incorporates Min-SNR-\(\gamma\) weighting:

\[\mathcal{L}_{\text{LDM}} = \min\left(\gamma, \frac{\bar{\alpha}_t}{1 - \bar{\alpha}_t}\right) \cdot \|\epsilon - \epsilon_\theta(z_t, t, \tau(y))\|_2^2\]

with \(\gamma=5\). This formulation caps the loss gradient precisely at low timesteps where the SNR is high, preventing the network from overfitting to high-frequency corruptions and prioritizing stable, low-frequency geometric identity structures.

3. ID-Preserving Sampling: Inference-Time Gradient-Guided Identity Correction

When pristine HQ references are extremely scarce (e.g., only 1 HQ reference amidst multiple LQ images), the target data distribution becomes ambiguous, leaving residual identity drift. To rectify this at inference without retraining or modifying model parameters, ID-PreFeR introduces test-time ID-preserving sampling. Utilizing a frozen ArcFace recognition model \(f_{\text{face}}\), the closest reference embedding \(f_{\text{face}}(x_{\text{ref}})\) is retrieved. At each denoising step, the predicted noise \(\hat{\epsilon}_t\) is refined via a gradient-ascent step on the cosine similarity between the decoded intermediate estimate \(\hat{x} = \mathcal{D}(\hat{z})\) and the reference embedding:

\[\epsilon^{(k+1)} = \epsilon^{(k)} + \delta \, \nabla_{\epsilon^{(k)}} \langle f_{\text{face}}(\mathcal{D}(\hat{z}(\epsilon^{(k)}))), f_{\text{face}}(x_{\text{ref}}) \rangle\]

Applied with step size \(\delta=0.05\) and \(K_{\text{ID}}=1\) exclusively during the final few reverse diffusion steps, this mechanism acts as an agile late-stage corrector, steering facial geometry and feature alignment toward the target identity while leaving the early-stage texture synthesis governed entirely by the diffusion prior.

Loss & Training

The overall training objective combines the weighted latent diffusion loss with the regularization prior loss: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{LDM}} + \lambda_{\text{reg}} \mathcal{L}_{\text{reg}}\)$ The network is personalized on SDXL with rank-16 LoRA on decoder cross-attention layers, trained using AdamW for 500 steps with a batch size of 4 in FP16 precision. The LoRA parameters use a square-root scaled learning rate of \(1 \times 10^{-3}\), while the learnable quality tokens \([Q]\) and identity token \([V]\) are trained with a learning rate of \(5 \times 10^{-4}\). Personalization requires approximately 4.2 minutes per subject.

Key Experimental Results

Main Results

On the CelebRef benchmark, ID-PreFeR was comprehensively evaluated against 4 blind face restoration baselines (GFPGAN, CodeFormer, DiffBIR, AuthFace) and 6 personalized/reference-based restoration methods (DMDNet, FaceMe, Gen2Res, InstantRestore, RestorerID, ReF-LDM).

Method Ref. spec. PSNRโ†‘ SSIMโ†‘ LPIPSโ†“ CLIPIQAโ†‘ MANIQAโ†‘ ID Scoreโ†‘
GFPGAN None (ร—) 25.50 0.748 0.301 0.503 0.449 0.613
CodeFormer None (ร—) 25.07 0.726 0.388 0.598 0.429 0.586
DiffBIR None (ร—) 25.29 0.690 0.401 0.615 0.504 0.562
AuthFace None (ร—) 25.32 0.683 0.294 0.699 0.643 0.649
DMDNet HQ only 25.04 0.712 0.316 0.613 0.491 0.635
FaceMe LQ & HQ 25.82 0.724 0.265 0.646 0.504 0.704
Gen2Res LQ & HQ 24.30 0.748 0.424 0.491 0.331 0.446
InstantRestore HQ only 24.65 0.722 0.297 0.561 0.488 0.769
RestorerID HQ only 22.72 0.636 0.434 0.717 0.574 0.522
ReF-LDM HQ only 23.58 0.676 0.244 0.669 0.562 0.695
Ours (ID-PreFeR) LQ & HQ 25.51 0.729 0.242 0.701 0.649 0.767
Input N/A 24.58 0.714 0.672 0.309 0.144 0.613
Ground truth N/A +โˆž 1.000 0.000 0.650 0.546 1.000

Note: Data from Table 1 of the paper. ID-PreFeR achieves the lowest LPIPS (0.242) and highest MANIQA (0.649), while reaching an ID score (0.767) on par with InstantRestore (0.769), despite using mixed LQ & HQ references rather than pure HQ inputs.

Ablation Study & Mixed-Quality Robustness

The ablation analysis below examines each constituent design and measures robustness across varying counts of high-quality references (#HQ \(\in \{5, 3, 1\}\)).

Config / Variant #Params PSNRโ†‘ SSIMโ†‘ LPIPSโ†“ CLIPIQAโ†‘ MANIQAโ†‘ IDโ†‘ (#HQ=5) IDโ†‘ (#HQ=3) IDโ†‘ (#HQ=1)
Complete Model (Ours) 6.41 M 25.51 0.729 0.242 0.701 0.649 0.7671 0.7641 0.7613
w/o prior preservation 6.41 M 25.52 0.731 0.245 0.699 0.624 0.7460 - -
w/o ID-quality disent. 6.41 M 25.57 0.698 0.279 0.674 0.555 0.7664 0.7631 0.7632
w/o ID-preserving sampling 6.41 M 25.07 0.696 0.281 0.666 0.554 0.7338 0.7325 0.7317
LoRA all CA 11.82 M 25.34 0.709 0.289 0.621 0.489 0.7450 - -
LoRA all CA & SA 12.57 M 25.45 0.720 0.290 0.599 0.472 0.7560 - -

Note: Compiled from Table 3 and Table 4 of the paper.

Key Findings

  • Robustness Under Extreme HQ Scarcity: When the number of HQ reference images drops from 5 to 1 (with degraded images filling the quota), ID-PreFeR maintains remarkably steady identity similarity (0.7671 \(\to\) 0.7641 \(\to\) 0.7613). Removing ID-preserving sampling results in an immediate drop across all counts to ~0.731, demonstrating that IDPS serves as the primary anchor for subject identity.
  • Disentanglement Drives Perceptual Clarity: Under #HQ=1, the variant without ID-quality disentanglement achieves an almost identical ID score (0.7632 vs. 0.7613) but suffers significant visual blurriness and artifact retention, with LPIPS degrading from 0.242 to 0.279 and MANIQA plummeting from 0.649 to 0.555. Disentanglement and Min-SNR-\(\gamma\) purify high-frequency textures rather than altering coarse facial geometry.
  • Lightweight Decoupled Adaptation Outperforms Full Attention Tuning: Expanding LoRA to all cross-attention and self-attention layers increases parameters to 12.57M yet degrades perceptual metrics (LPIPS drops to 0.290, CLIPIQA falls to 0.599), confirming that targeted decoder cross-attention injection prevents feature distortion while maintaining prompt responsiveness.

Highlights & Insights

  • MLLM-Guided Textual Quality Decoupling: Leveraging a lightweight MLLM to assign binary quality tokens and initializing them with semantic opposites (sharp/blurry) elegantly absorbs degradation patterns into auxiliary tokens, leaving the identity token pure and transferrable at inference.
  • SNR Analysis Unveils Degradation Overfitting: By deriving the forward diffusion equations with latent degradation residuals, the paper rigorously pinpoints low-timestep, high-SNR regimes as the primary culprit behind artifact memorization, curing it with clean Min-SNR-\(\gamma\) clamping.
  • Practical 0.25% Footprint for Consumer Scenarios: Tuning and storing only 6.41M parameters (a 13MB file) in 4.2 minutes provides an economically viable path for personal mobile photo restoration compared to previous multi-gigabyte models.

Limitations & Future Work

  • Feature Cancellation Across Accessory Discrepancies: Inconsistent makeup, glasses, and jewelry across diverse references tend to be averaged out, causing intermittent disappearance or structural uncertainty in auxiliary attributes.
  • Increased Test-Time Latency from Gradient Guidance: Backpropagating similarity gradients through the VAE decoder and ArcFace network increases single-image inference latency from 5.6s to 29.0s and memory from 24GB to 34.8GB, presenting challenges for real-time mobile deployment.
  • vs DMDNet / FaceMe: Traditional exemplar-based methods (e.g., DMDNet) suffer from structural misalignment under severe pose variation, while contemporary diffusion models (e.g., FaceMe) degrade steeply under mixed reference quality. ID-PreFeR achieves superior consistency and image fidelity across varied inputs.
  • vs DreamBooth / Textual Inversion: Classic personalization relies heavily on thousands of regularization images to prevent language drift. ID-PreFeR demonstrates that a simple latent prior loss \(\mathcal{L}_{\text{reg}}\) referenced against the frozen backbone is sufficient to stabilize generation with zero external regularization images.

Rating

  • Novelty: โญโญโญโญโ˜† Intuitive integration of MLLM quality tokens, SNR-capped diffusion loss, and late-stage test-time gradient refinement.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluated across synthetic CelebRef, large-scale FFHQ-Ref-Moderate, and newly collected in-the-wild mobile captures with comprehensive user/VLM evaluations.
  • Writing Quality: โญโญโญโญโญ Cohesive theoretical narrative connecting latent degradation formulations directly to empirical design choices.
  • Value: โญโญโญโญโญ Directly resolves the ubiquitous real-world challenge of imperfect, mixed-quality reference photos in consumer-facing portrait restoration.