Anchoring on Reality: Breaking the Pseudo-Target Ceiling in Makeup Transfer

Conference: ECCV 2026
arXiv: 2606.31089
Project Page: https://csbowei.github.io/ART/
Code: None
Area: Image Generation
Keywords: Makeup Transfer, Diffusion Transformer, Reality Anchoring, Differentiable Reconstruction, Pseudo-Target Ceiling, High-Resolution Dataset

TL;DR

Proposes the ART two-stage DiT framework, which breaks the pseudo-target ceiling through a differentiable reality anchoring loop—reconstructing the real reference from a bare-skin reference to allow weight gradients to penalize omissions—achieving high-fidelity transfer of complex makeup such as dense stickers and face painting at 2K resolution.

Background & Motivation

The goal of makeup transfer is to apply the makeup style of a reference image to a source face image while maintaining the source identity and geometry. In addition to daily beauty enhancement, it is also critical in scenarios such as film and television post-production and virtual digital humans—scenarios that involve complex artistic styles like dense stickers, face painting, and glitter, and often encounter cross-gender and cross-ethnic transfer requirements. However, this task faces a fundamental data bottleneck: precisely aligned images of the same person with and without makeup are almost impossible to obtain.

Existing methods take two paths to bypass this data bottleneck. The first is the GAN-based weak prior route, which relies on cycle consistency or domain adversarial loss for unpaired mapping (CycleGAN, LADN, CSD-MT, etc.). These methods exhibit limited performance in fine-grained local placement and photo-realism, often degrading into global color overlays when facing fine textures such as glitter and stickers. The other route utilizes large-scale image editing models (e.g., Nano Banana Pro, StableMakeup) to synthesize pseudo-paired data, allowing models to directly regress toward pseudo-targets. This significantly improves overall realism but introduces an intrinsic pseudo-target ceiling—the model systematically inherits the biases and artifacts in the pseudo-targets, including identity drift, detail degradation, and unnecessary background editing. More importantly, since the training target is only to passively imitate pseudo-targets, the model theoretically can never surpass the quality upper bound of the pseudo-targets.

The core insight of this paper is: although paired transfer targets are unavailable, the real reference image itself already contains all the required makeup details—the model does not need to copy the patterns of pseudo-targets, but should extract cosmetic details itself during the process of reconstructing the real reference. Core Idea: Shift makeup transfer from regressing toward synthetic pseudo-targets to differentiable reality-anchored reconstruction. Through a two-stage DiT framework, a differentiable loop reconstructing the real reference from the bare-skin reference is constructed in Stage II, allowing reconstruction gradients to backpropagate through the DiT to the early transfer predictions, directly penalizing the structures of omitted cosmetics and forcing the model to actively extract cosmetic textures and cover pseudo-target artifacts.

Method

Overall Architecture

This paper proposes ART (Anchoring on Reality Makeup Transfer), a two-stage training framework based on Diffusion Transformer (DiT), utilizing the backend FLUX.1-Kontext-dev as the base model and fine-tuning with LoRA. The overall pipeline is divided into two stages: Stage I initializes with pseudo-targets, allowing the model to acquire basic semantic alignment and global makeup placement capabilities by learning to imitate synthetic pseudo-targets, while simultaneously training an auxiliary makeup remover \(R\) to extract the bare-skin reference from the reference image; Stage II switches the supervision signal from pseudo-targets to the real reference, forcing the model to repair fine-grained makeup textures through a differentiable reconstruction loop.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    subgraph S1["Stage I: Pseudo-Target Initialization"]
        A["Source Image I_src<br/>Reference Image I_ref"] --> B["Transfer DiT<br/>LoRA Flow Matching Training"]
        B --> C["Pseudo-Target Imitation<br/>Global Alignment + Basic Makeup Placement"]
    end

    subgraph S2["Stage II: Reality-Anchored Refinement"]
        C -->|Pseudo-target latent z_pseudo| D["Add Noise z_σtr<br/>σtr=0.6"]
        D --> E["One-step Euler Prediction<br/>ẑ = z_σtr - σtr·vθ"]
        E -->|Differentiable Carrier ẑ| F["Reconstruct Real Reference z_ref<br/>Condition: z_bare_ref + ẑ"]
        F --> G["L_refine Penalizes Omissions<br/>Gradient Backpropagation"]
        G -->|Update Velocity Field| E
    end

    A --> H["Makeup Remover R<br/>Bare Skin Extraction"]
    H --> I["Bare-skin Reference z_bare_ref"]
    I --> F

    D -.-> J["L_bottleneck<br/>Structural Regularization"]
    J -.-> E

    E --> K["Inference: Single Forward Pass<br/>28-step Euler Sampling"]

Crucial input-output relationships: The inputs are the source image \(I_{src}\) and reference image \(I_{ref}\), and the output is the generated image that preserves the source identity while transferring the reference makeup. The system operates in the VAE latent space and trains a conditional DiT using the Flow Matching objective. During inference, only a single forward pass is required, with no need for pseudo-targets, makeup removers, or iterative optimization.

Key Designs

1. Two-Stage Progressive Training: First Imitate, Then Surpass

In Stage I, the main transfer model and the makeup remover \(R\) are trained independently. The transfer model is initialized with the standard Flow Matching loss on pseudo-targets, where the conditional encoding is the latent variables of the source and reference images. The goal is for the model to learn basic facial structure alignment and global makeup placement. The makeup remover \(R\) is a one-step denoising model trained under the supervision of \((I_{pseudo}, I_{src})\), with an additional perceptual loss (\(L_{lpips}\)), identity consistency loss (ArcFace embedding), and landmark geometric loss to guarantee the structural fidelity of bare-skin extraction. Stage II is the core innovation—the supervision signal is switched from pseudo-targets to the real reference, and refined through a differentiable reconstruction loop. Ablation studies show that both stages are indispensable: using only Stage I leads to identity drift (the model is satisfied with imitating flawed pseudo-targets), while using only Stage II results in an uninformative makeup carrier and a reconstruction signal that is too weak; the two stages complement each other to achieve both high makeup fidelity and identity preservation.

2. Differentiable Cosmetics Carrier and Reconstruction Loop

This is the fundamental mechanism for breaking the pseudo-target ceiling. Given the pseudo-target latent \(z_{pseudo}\) obtained in Stage I, noise is first added at the noise level \(\sigma_{tr}\) to obtain \(z_{\sigma_{tr}}\), followed by a one-step Euler prediction to obtain the cosmetics carrier \(\hat{z} = z_{\sigma_{tr}} - \sigma_{tr} \cdot v_{\theta}(z_{\sigma_{tr}}, \sigma_{tr}; z_{src}, z_{ref})\). Crucially, \(\hat{z}\) is not decoded into a pixel image but is kept as a differentiable tensor in the computation graph. For the same identity, \(\hat{z}\) and the bare-skin reference \(z_{bare\_ref}\) are used as conditions to reconstruct the real reference \(z_{ref}\) through the same DiT model sharing weights. Because \(\hat{z}\) is differentiable, the gradient of the reconstruction loss \(L_{refine}\) can backpropagate through the DiT to the initial velocity field prediction that produced \(\hat{z}\). This means that if the initial transfer prediction misses any cosmetic details (e.g., if a patch of glitter is not applied), reconstructing the real reference will fail, \(L_{refine}\) will increase, and the gradient will naturally guide \(\hat{z}\) to capture more reference textures. This is fundamentally different from traditional cycle consistency: CycleGAN only verifies whether it can go back to the source image (domain-level regularization), whereas this verifies whether the real reference can be reconstructed (instance-level supervision). The former ensures bidirectional consistency, while the latter directly demands the inclusion of all reference cosmetic details. A clever byproduct of this design is that the slight blurriness and residuals of the makeup remover \(R\) act as an implicit regularization—forcing \(\hat{z}\) to only encode high-frequency cosmetic textures and prevent it from encoding identity information.

3. Controlled Noise Bottleneck: Dual Balancing of \(\sigma_{tr}\) and \(L_{bottleneck}\)

\(\sigma_{tr}\) is the only but critical hyperparameter in the entire framework, determining how much prior information is inherited from the pseudo-target. When \(\sigma_{tr}\) is small (e.g., 0.2), \(\hat{z}\) is tightly constrained near the pseudo-target, leading to conservative editing and the retention of pseudo-target artifacts. When \(\sigma_{tr}\) is large (e.g., 0.8), the generation has higher degrees of freedom and better detail recovery, but the structural prior is too weak, potentially causing identity drift. The paper systematically ablates \(\sigma_{tr}\) between 0.2 and 0.8 to identify \(\sigma_{tr}=0.6\) as optimal. Meanwhile, \(L_{bottleneck}\) is introduced as a structural regularization term: it constrains the consistency of the velocity field output with the pseudo-target prior at the noise level \(\sigma_{tr}\). If there were only \(L_{refine}\) without \(L_{bottleneck}\), the model would take a shortcut—directly ignoring the source image identity and encoding \(\hat{z}\) as the entire reference image to minimize the reconstruction loss (i.e., outputting a copy of the reference image). \(L_{bottleneck}\) anchors the global structure corresponding to the source identity, forcing \(\hat{z}\) to only extract cosmetic textures without encoding identity. \(\sigma_{tr}\) and \(L_{bottleneck}\) form a dual balance: the former controls the extent of detail recovery, while the latter prevents shortcut degradation solutions.

Loss & Training

Stage I: Pseudo-Target Initialization - Transfer model: \(L_{init} = L_{FM}(\theta; z_{pseudo}, z_{src}, z_{ref})\), standard Flow Matching loss - Makeup remover \(R\): \(L_R = L_{FM}(\phi; z_{src}, z_{pseudo}) + \lambda_{lpips} \cdot L_{lpips} + \lambda_{id} \cdot L_{id} + \lambda_{lmk} \cdot L_{lmk}\)

Stage II: Reality-Anchored Refinement - Total objective: \(L_{s2} = L_{refine} + 0.2 \cdot L_{bottleneck}\) - \(L_{refine} = L_{FM}(\theta; z_{ref}, z_{bare\_ref}, \hat{z})\), reconstructing the real reference - \(L_{bottleneck} = \mathbb{E}[w(\sigma_{tr}) \|v_{\theta}(z_{\sigma_{tr}}, \sigma_{tr}; z_{src}, z_{ref}) - (\epsilon - z_{pseudo})\|^2]\), structural regularization - \(\sigma_{tr} = 0.6\)

Training Configurations: Base model FLUX.1-Kontext-dev, LoRA rank \(r=32\), scaling factor 32. Optimizer Prodigy (lr=1), 4× NVIDIA H20 GPUs, batch size = 16 at 512 resolution, 4 at 2K resolution. Stage I trains for 10K steps, and Stage II trains for another 10K steps. Additional high-resolution refinement can incorporate wavelet loss \(L_{WLF}\) to enhance high-frequency details. Makeup remover \(R\) is frozen during Stage II.

Inference: Requires only a single forward pass of the transfer model (28-step Euler sampling), with no need for pseudo-targets, makeup removers, or iterative loops.

Key Experimental Results

Main Results

Evaluated on four datasets (MT, LADN, MT-Wild, MF2K) across four dimensions: makeup similarity (VLM scores MSimG/MSimQ), identity preservation (ArcFace ID), background stability (L2-M), and image quality (FID).

Dataset	Metric	ART	StableMakeup	GPT Image 1.5	Banana Pro
MF2K (Artistic Makeup)	MSimG↑	9.22	6.73	8.43	8.34
MF2K	ID↑	0.74	0.43	0.35	0.65
MF2K	L2-M↓	4.31	9.26	28.32	13.27
MT	MSimG↑	8.96	8.34	8.69	8.34
MT	ID↑	0.87	0.49	0.36	0.67
LADN	MSimG↑	9.14	7.81	8.85	8.18

ART achieves an MSimG of 9.22 on MF2K (outperforming the second-place GPT 8.43 by 0.79 points), the lowest background instability L2-M overall (4.31), and the highest identity preservation ID on MT (0.87). In the user study (21 participants, 864 ratings), ART ranks first across all three metrics, with an identity consistency score of 4.03/5, leading significantly.

Ablation Study

Configuration	MF2K MSimG↑	MF2K ID↑	MF2K L2-M↓	Explanation
Stage I Only (Pseudo-Targets)	8.82	0.57	7.68	Severe identity drift
Stage II Only (Refinement)	8.08	0.70	4.98	Cosmetics carrier is uninformative
Full (I + II)	9.22	0.74	4.31	The two stages complement each other
W/o L_bottleneck	—	Degenerated solution (directly copies reference identity)	—	Shortcut guardrail
\(\sigma_{tr}=0.2\)	~8.2	~0.72	~4.2	Retains pseudo-target artifacts
\(\sigma_{tr}=0.6\) (Optimal)	9.22	0.74	4.31	Best balance point
\(\sigma_{tr}=0.8\)	~8.9	~0.65	~5.5	Identity drift

Key Findings

• \(\sigma_{tr}\) is the most sensitive hyperparameter, controlling the trade-off between prior preservation and detail recovery, with \(\sigma_{tr}=0.6\) being optimal.
• The impact of pseudo-target quality is limited: Even starting from weaker StableMakeup pseudo-targets (Ours_SM), the refined MSimG of 8.56 still far exceeds StableMakeup's own 6.73—demonstrating that reality-anchored refinement can systematically surpass pseudo-targets.
• An additional ~4,500 unpaired images without pseudo-targets participated in Stage II refinement, boosting MSimG on MF2K from 9.14 to 9.22.
• The slight blurriness of makeup remover \(R\) is a feature, not a defect—it forces \(\hat{z}\) to focus on high-frequency makeup textures, achieving implicit identity and makeup decoupling.
• ART demonstrates zero-shot generalization capabilities across cross-domain scenarios (anime, 3D avatars).
• Limitations: It does not explicitly model physical lighting consistency, which may produce contradictory highlights when light source directions are severely mismatched.

Highlights & Insights

• A new perspective to bypass the pseudo-target ceiling: While prior work focused entirely on making outputs resemble pseudo-targets, this paper elegantly argues that "although transfer targets are unavailable, the real reference itself is the best supervision," shifting the objective from "generating as well as pseudo-targets" to "generating what can reconstruct the reference." This paradigm shift is highly critical.
• Simplicity of the differentiable loop: It avoids introducing extra generation branches or discriminators, simply maintaining \(\hat{z}\) in a differentiable computation graph and reusing the same DiT for reconstruction to close the gradient feedback loop. At inference, the loop is completely bypassed, ensuring no impact on efficiency.
• The noise bottleneck as a dual-function control valve: \(\sigma_{tr}\) not only regulates the amount of pseudo-target prior retained but also prevents identity degradation via \(L_{bottleneck}\). A single hyperparameter simultaneously satisfies the two requirements of maintaining structure and restoring details, yielding a clean design.
• Inspiration for general unpaired editing tasks: The differentiable reconstruction loop is not limited to makeup transfer—unpaired image editing tasks lacking ground truth, such as virtual try-on, hairstyle transfer, and stylization, can all benefit from this paradigm.

Limitations & Future Work

• Physical consistency of illumination and reflections is not explicitly modeled—the physical mismatch of glitter and metallic textures between strong directional light and soft environmental light is not addressed yet. A physical lighting decomposition module could be integrated in the future.
• Higher training cost: Stage II requires an additional reconstruction forward pass (approx. 1.6× time) and relies on commercial models (Banana Pro) for high-quality pseudo-target generation.
• Trained only on real face data; although it exhibits zero-shot generalization on anime and 3D avatars, systematic performance on out-of-distribution scenarios remains to be validated.
• The makeup remover \(R\) can be upgraded: currently a one-step denoising model, it could be replaced with multi-step diffusion in the future for cleaner bare-skin anchors, potentially further enhancing refinement quality.

Related Work & Insights

• vs StableMakeup: StableMakeup directly regresses toward pseudo-targets, capped by the ceiling; ART breaks through this ceiling using a reality-anchored loop, outperforming StableMakeup even when initialized on lower-quality pseudo-targets.
• vs CycleGAN: Traditional cycle consistency only verifies the ability to return to the source, acting as a domain-level regularization (unable to guarantee instance-level details); ART reconstructs the real reference, providing instance-level supervision and directly requiring complete cosmetic details.
• vs SHMT/MAD: SHMT uses self-supervised hierarchical decoupling, and MAD uses domain-unified embeddings, both of which still operate within a weak prior framework; ART's differentiable reconstruction loop provides a level of granularity that transcends weak priors.
• General insight for image editing: "When paired data is unavailable, use the reference itself to construct a differentiable reconstruction loop"—this paradigm offers strong inspiration for virtual try-on, hairstyle transfer, and style transfer.

Rating

• Novelty: ⭐⭐⭐⭐⭐ "Shifting from pseudo-targets to differentiable reality-anchored reconstruction" is a new paradigm in makeup transfer, and the noise bottleneck design is elegant.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 4 datasets + 864 user trials + systematic ablation studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-argued motivations, clear architecture, and substantial supplementary materials.
• Value: ⭐⭐⭐⭐⭐ Solves a long-standing core bottleneck in makeup transfer; the methodology exhibits great potential for cross-task adaptation.

Related Papers

• [CVPR 2026] Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features
• [ECCV 2026] Learn Once, Edit Anywhere: Visual Direction Transfer for Diffusion Models
• [ICLR 2026] ContextGen: Contextual Layout Anchoring for Identity-Consistent Multi-Instance Generation
• [ECCV 2026] H-Adapter: Pose-Robust Hairstyle Transfer via Attention-Derived, Source-Aligned Hair Masks
• [CVPR 2026] Anchoring and Rescaling Attention for Semantically Coherent Inbetweening