title: >- [Paper Note] MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfer description: >- [ECCV 2026][Others] MagicMakeup achieves high-fidelity, region-controllable makeup transfer via Token-Aligned Region Gating (TARG), Cross-Modal Perception Guidance (CMPG), and a 1024x1024 makeup-removal data pipeline. tags: - ECCV 2026 - Others - Diffusion Transformer - Makeup Transfer - Region Control date: 2026-09-19 content_hash: 07cac295e6339d37
MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfer¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://vivocameraresearch.github.io/magicmakeup
Area: Others
Keywords: Region-Controllable Makeup Transfer, Diffusion Transformer, Attention Gating, Cross-Modal Disentanglement, High-Resolution Benchmark
TL;DR¶
To tackle cross-region makeup spillover, concept confusion between source identity and reference makeup, and scarce paired supervision, MagicMakeup introduces a Diffusion Transformer framework with Token-Aligned Region Gating and Cross-Modal Perception Guidance alongside a 1024×1024 regional makeup-removal data construction pipeline for high-fidelity facial makeup transfer.
Background & Motivation¶
Facial makeup transfer aims to apply cosmetic styles, textures, and colors from a reference makeup image to a source face while faithfully preserving the source identity, facial geometry, and background context. Driven by consumer-grade virtual try-on, beauty cameras, and digital avatars, current requirements have transitioned from global, coarse full-face color shifts to fine-grained regional control (such as editing eyeshadow or lipstick independently) and mixing multiple references at ultra-high resolution. However, existing generative adversarial network (GAN) and diffusion-based approaches struggle to satisfy these criteria simultaneously due to three compounding obstacles.
The first core obstacle is limited spatial controllability caused by pixel-to-attention misalignment. Most diffusion models operate at the full-face scale; even when spatial pixel masks are supplied, downsampling operations and unconstrained attention receptive fields allow cosmetic feature tokens to disperse into adjacent areas, leading to color bleeding on cheeks and brows. The second hurdle is conceptual ambiguity under two-image conditioning: models easily confuse the preservation of source identity with the transfer of reference style, inadvertently leaking the reference subject's bone structure or facial geometry into the output, or conversely diluting the cosmetic intensity to protect identity. Lastly, authentic paired supervision is virtually nonexistent due to inevitable variations in lighting, expression, and head pose. Synthetic pseudo-pairs created by text-guided makeup application fail to capture high-frequency makeup textures, distort facial structure, and introduce severe distribution shift.
To break this bottleneck, the authors reverse the conventional synthetic paradigm by performing region-specific makeup removal on real makeup portraits, yielding pristine paired supervision while directly enforcing spatial and conceptual constraints inside the Multi-Modal Diffusion Transformer (MM-DiT). Core idea: project region masks directly into the attention logit space via Token-Aligned Region Gating to shut off cross-region attention leakage, coupled with Cross-Modal Perception Guidance to semantically decouple preservation and transfer concepts across modalities.
Method¶
Overall Architecture¶
MagicMakeup is built upon the Multi-Modal Diffusion Transformer (MM-DiT) architecture. Its processing pipeline takes four inputs: a source face image \(\mathbf{I}_s\), a reference makeup portrait \(\mathbf{I}_r\), refined region masks \(\mathbf{M}_s\) and \(\mathbf{M}_r\), and an editing instruction text prompt \(\mathbf{T}\). Visual inputs are mapped by frozen vision encoders into token sequences \(\mathbf{X}_s\) and \(\mathbf{X}_r\), while the generation process initiates from Gaussian latent noise \(\mathbf{X}_n\). Text instruction \(\mathbf{T}\) is encoded into text embeddings \(\mathbf{X}_T\). During each iterative denoising step, Token-Aligned Region Gating (TARG) applies logit masks onto latent-to-reference token interactions, while Cross-Modal Perception Guidance (CMPG) modulates image and text representations to strictly isolate identity preservation cues from cosmetic attributes. Finally, an image decoder synthesizes the denoised latent representation into the high-resolution output \(\mathbf{I}_o\).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Inputs: Source Is, Reference Ir, Masks Ms/Mr, Instruction T"] --> B["Token-Aligned Region Gating (TARG)<br/>Project pixel masks to logit gates to block cross-region leakage"]
B --> C["Cross-Modal Perception Guidance (CMPG)<br/>Decouple transfer and preservation concepts across modalities"]
C --> D["Real Makeup Removal Pipeline<br/>Construct identity-consistent pairs via local de-makeup on real photos"]
D --> E["MM-DiT Iterative Denoising and High-Res Decoding<br/>Synthesize region-controlled output Io preserving source geometry"]Key Designs¶
1. Token-Aligned Region Gating (TARG): Aligning Pixel Masks to Attention Logits to Suppress Cross-Region Bleeding Standard spatial editing approaches that apply masks in pixel space or concatenate masked images fail in multi-head cross-attention because tokens interact globally across the entire spatial extent. To resolve this, TARG translates spatial region masks into the attention domain. Source and reference masks \(\mathbf{M}_s\) and \(\mathbf{M}_r\) for the target region (face, eyes, or lips) are downsampled and flattened to match the token dimensions of MM-Attention, yielding a binary latent query gate \(\mathbf{m}_q \in \{0, 1\}^{L_n}\) and a reference key gate \(\mathbf{m}_k \in \{0, 1\}^{L_r}\). An additive logit gating matrix \(\mathbf{M}\) is defined across latent query tokens \(i\) and reference key tokens \(j\):
This matrix modifies the scaled dot-product attention calculation: \(\text{Attn}(Q, K, V) = \text{Softmax}\left(\frac{QK^T}{\sqrt{d_k}} + \mathbf{M}\right)V\). Because \(\mathbf{M}_{i,j}\) takes a large negative constant whenever the query token lies outside the region of interest, the attention weight assigned to the reference makeup image collapses to zero. Consequently, non-target areas (such as cheeks during an eye edit, or surrounding hair and background) never query the reference tokens, eliminating color bleeding, unwanted face reshaping, and background warping.
2. Cross-Modal Perception Guidance (CMPG): Bidirectional Semantic Decoupling of Preservation and Transfer Concepts Conditioning on two separate facial images often induces conceptual confusion in DiT models, where the network conflates the reference identity with cosmetic attributes. CMPG extracts two explicit concept descriptors from the text instruction \(\mathbf{T}\): a preservation concept \(\mathbf{C}_s\) (describing identity, structural fidelity, and non-target facial cues) and a transfer concept \(\mathbf{C}_r\) (specifying the cosmetic product, color, and texture, such as lipstick or eyeshadow). For each concept-image pair \((\mathbf{C}_s, \mathbf{I}_s)\) and \((\mathbf{C}_r, \mathbf{I}_r)\), concept text tokens \(\mathbf{X}_C\) and visual tokens \(\mathbf{X}_I\) from deep layers of frozen backbones are co-updated via joint self-attention to align semantics, yielding \(\tilde{\mathbf{X}}_C\) and \(\tilde{\mathbf{X}}_I\). To focus updates exclusively on discriminative evidence, visual tokens are ranked by their similarity to concept tokens to select top-\(k\) visual queries \(\mathbf{Q}\):
These queries \(\mathbf{Q}\) interact via dual cross-attentions with a multi-layer visual context bank (extracted from the last three backbone layers) and concept text tokens. Adaptive Layer Normalization (AdaLayerNorm) heads then map the refined features into span-aware text modulations \(\mathbf{X}_{C_s}^*, \mathbf{X}_{C_r}^*\) and image modulations \(\mathbf{X}_{I_s}^*, \mathbf{X}_{I_r}^*\). The text modulations update only the specific token spans of \(\mathbf{C}_s\) and \(\mathbf{C}_r\) in \(\mathbf{X}_T\), while image modulations are injected into the respective DiT image streams through cross-attention adapters. This dynamic mechanism ensures that at every denoising timestep, the model explicitly attributes identity to the source stream and cosmetic style to the reference stream.
3. Real Makeup Removal Pipeline: Sourcing High-Fidelity Training Supervision via Inverse De-Makeup Synthesizing makeup onto bare faces using generative models typically introduces blurry cosmetic boundaries and severe identity shifts. In contrast, removing makeup from an existing portrait is mathematically and geometrically better constrained: de-makeup operates as the subtraction of an appearance layer while retaining the subject's authentic underlying anatomy and realistic skin texture. Starting from a collection of 50,000 real \(1024 \times 1024\) makeup portraits, the authors filter out low-resolution or off-angle faces, retaining 3,000 pristine images grouped into full-face, eye, and lip splits. FLUX-Kontext performs region-specific makeup removal to generate bare-face source candidates.
To ensure immaculate alignment, a four-stage filtering cascade is enforced: - Geometry & Deformation Filter: DFA-MobileNet aligns facial keypoints; AdaFace cosine similarity discards distorted or under-edited candidates; landmark contour differences eliminate eyelid state shifts (blinking) or mouth posture differences (teeth exposure). - Residual Makeup Filter: Local color-histogram divergence guarantees complete cosmetic removal; a composite score balancing identity similarity and regional color difference selects the optimal candidate per group. - Precision Region Filter: Enforces large color discrepancies inside the target region alongside minimal variation across non-target areas, eliminating global tone shifts. This yields 6,772 clean, region-labeled pairs (2,392 face, 2,224 eye, 2,156 lip). Furthermore, to synthesize reference portraits with decoupled identities, makeup faces are warped onto target faces of markedly different identities using 478 MediaPipe facial landmarks, Delaunay triangulation, and piecewise-affine transformations, preventing structural leakage from the reference image.
Loss & Training¶
The architecture is trained on 8 NVIDIA A100 80GB GPUs with a batch size of 2 per GPU. Optimization is conducted using AdamW at an initial learning rate of \(5 \times 10^{-5}\) for 36,000 iterations. To maintain generative priors and preserve training efficiency, DiT weights are tuned via Low-Rank Adaptation (LoRA) with rank \(r = 256\). Training text prompts are constrained to predefined categories. During inference, sampling runs for 28 steps under a Classifier-Free Guidance (CFG) scale of 2.5.
Key Experimental Results¶
Main Results¶
Evaluation is performed across three benchmarks: two \(1024 \times 1024\) high-resolution datasets (MakeupHQ-Synthetic and MakeupHQ-Real, 500 pairs each) and the public \(256 \times 256\) Makeup-Wild benchmark (1,000 pairs). Evaluation metrics include DINO-I and CLIP-I for cosmetic transfer fidelity, AdaFace cosine similarity for Face-ID preservation, L2M for mean squared error across non-edited background regions, and Inception-based FID for visual realism.
| Dataset | Metric | CPM [26] | EleGANt [42] | SSAT [34] | CSD-MT [36] | SHMT [35] | MAD [31] | Stable-Makeup [44] | Flux-Makeup [48] | Ours |
|---|---|---|---|---|---|---|---|---|---|---|
| MakeupHQ-Synthetic (\(1024 \times 1024\)) |
DINO-I ↑ CLIP-I ↑ Face-ID ↑ L2M ↓ FID ↓ |
0.625 0.777 0.400 1610.94 72.19 |
0.520 0.744 0.796 2148.09 86.84 |
0.461 0.698 0.684 2816.61 120.19 |
0.576 0.796 0.563 3334.56 83.90 |
0.488 0.710 0.824 823.66 92.60 |
0.500 0.727 0.590 180.67 91.37 |
0.612 0.789 0.569 101.42 69.73 |
0.576 0.765 0.806 111.48 64.12 |
0.660 0.816 0.730 56.37 59.24 |
| MakeupHQ-Real (\(1024 \times 1024\)) |
DINO-I ↑ CLIP-I ↑ Face-ID ↑ L2M ↓ FID ↓ |
0.575 0.722 0.361 2158.13 99.11 |
0.534 0.733 0.755 4038.09 97.25 |
0.448 0.665 0.641 3138.46 131.55 |
0.596 0.772 0.522 5572.25 91.93 |
0.482 0.682 0.777 593.03 115.32 |
0.476 0.703 0.539 183.09 119.16 |
0.616 0.774 0.525 163.98 81.34 |
0.552 0.745 0.792 104.46 85.43 |
0.623 0.799 0.706 49.77 76.27 |
| Makeup-Wild (\(256 \times 256\)) |
DINO-I ↑ CLIP-I ↑ Face-ID ↑ L2M ↓ FID ↓ |
0.608 0.621 0.406 605.86 86.59 |
0.587 0.640 0.777 1540.74 94.20 |
0.569 0.581 0.706 2709.62 109.18 |
0.602 0.640 0.499 2151.73 94.71 |
0.564 0.586 0.847 1246.03 103.27 |
0.545 0.609 0.598 169.28 105.79 |
0.619 0.637 0.721 71.53 85.13 |
0.568 0.621 0.928 42.61 88.71 |
0.581 0.665 0.884 25.84 92.92 |
Ablation Study¶
Ablation experiments conducted on MakeupHQ-Synthetic across eye, lip, and full-face editing tasks validate the individual contributions of TARG and CMPG:
| Region | Config | TARG | CMPG | Face-ID ↑ | DINO-I ↑ | CLIP-I ↑ | Note |
|---|---|---|---|---|---|---|---|
| Eyes | Variant 1 Variant 2 Variant 3 |
× ✓ ✓ |
× × ✓ |
0.870 0.890 0.922 |
0.791 0.799 0.801 |
0.883 0.890 0.904 |
Base DiT model +TARG blocks leakage into lips/cheeks, boosting Face-ID Full model: CMPG refines eyeshadow textures and semantic alignment |
| Lip | Variant 1 Variant 2 Variant 3 |
× ✓ ✓ |
× × ✓ |
0.879 0.885 0.971 |
0.653 0.659 0.673 |
0.877 0.870 0.884 |
Base DiT model +TARG restrains lipstick to vermilion borders Full model: Lip texture and specular highlight transfer optimized |
| Face | Variant 1 Variant 2 Variant 3 |
× ✓ ✓ |
× × ✓ |
0.805 0.857 0.858 |
0.733 0.712 0.753 |
0.833 0.811 0.840 |
Base DiT model +TARG prevents background warping and facial distortion Full model: Best balance between cosmetic coverage and facial structure |
Key Findings¶
- Order-of-Magnitude Reduction in Non-Target Distortion: On both \(1024 \times 1024\) datasets, prior methods exhibited high L2M values (101.42–5572.25) due to global face rescaling and background shifting. MagicMakeup drops L2M to 56.37 on Synthetic and 49.77 on Real, cutting distortion in half compared to the nearest diffusion baselines (Stable-Makeup and Flux-Makeup).
- Optimal Pareto Trade-off Between Makeup Transfer and Identity Preservation: While certain baselines (e.g., Flux-Makeup) exhibit marginally higher Face-ID scores, visual inspection confirms that they transfer very conservative, faint makeup shades (evidenced by substantially lower DINO-I scores of 0.576 and 0.552). MagicMakeup delivers the highest DINO-I (0.660 / 0.623) and CLIP-I (0.816 / 0.799) while maintaining strong identity fidelity.
- Decoupled Architectural Roles: TARG acts as an explicit spatial filter that directly improves identity preservation (Face-ID) by restricting cross-attention to target ROIs. CMPG acts as a semantic amplifier that boosts cosmetic fidelity (DINO-I, CLIP-I) by guiding the latent representation to absorb cosmetic nuances without structural leakage.
Highlights & Insights¶
- Attention Logit Masking for Spatial Containment: Rather than post-processing with Alpha blending or relying on unconstrained cross-attention, injecting binary logit masks directly into DiT cross-attention blocks completely severs reference token leakage into unedited regions at zero inference overhead.
- De-Makeup Pairing Paradigm: Overcoming the fundamental distribution shift and artifacts of generative makeup application by performing targeted makeup removal on authentic high-resolution portraits provides an elegant, scalable data construction blueprint.
- Dynamic Cross-Modal Concept Guidance: Splitting complex multimodal editing prompts into explicit preservation and transfer concepts and routing them via AdaLayerNorm modulations prevents identity-makeup entanglement across diffusion trajectories.
Limitations & Future Work¶
- Author-Acknowledged Limitations: The framework currently provides regional control over eyes, lips, and full-face regions, but does not yet extend to individual sub-attributes like blush, eyebrow styling, or facial contouring. In addition, cosmetic transfer fidelity may degrade on extreme profile angles (\(> 60^\circ\) yaw) due to a scarcity of extreme-pose samples in real-world data.
- Identified Weaknesses: The de-makeup data pipeline depends on FLUX-Kontext; any subtle skin texture smoothing introduced during inpainting could inject a mild softening prior. Furthermore, soft eyeshadow masks require heuristic radial gradients and dilation, leaving boundary transitions sensitive to landmark accuracy.
- Future Directions: Exploring differentiable soft attention gating for seamless blending of blush and contouring, as well as incorporating 3D face mesh models to resolve extreme-pose self-occlusion.
Related Work & Insights¶
- vs Stable-Makeup [44]: Stable-Makeup employs CLIP-guided conditioning and facial control modules but lacks token-level attention gating, leading to subtle background stretching and boundary bleeding; MagicMakeup eliminates spillover through TARG, cutting non-target L2M error by over 50%.
- vs MAD [31]: MAD introduces cross-domain diffusion for makeup manipulation but suffers from conceptual ambiguity and high memory consumption at high resolutions; MagicMakeup utilizes MM-DiT with CMPG to achieve crisp conceptual separation and efficient \(1024 \times 1024\) synthesis.
- vs Flux-Makeup [48]: Flux-Makeup trains DiTs on synthetic makeup-applied pairs, resulting in conservative color shifts to guard identity; MagicMakeup leverages authentic de-makeup pairs and affine landmark warping, achieving superior cosmetic texture fidelity without compromising facial structure.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ (Attention logit gating and de-makeup pairing pipeline offer clean, effective solutions to longstanding spatial control issues)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Extensive quantitative benchmarking across 1024 synthetic/real splits and 256 in-the-wild data with granular ablations)
- Writing Quality: ⭐⭐⭐⭐⭐ (Precise problem framing, rigorous mathematical definitions, and lucid visual illustrations)
- Value: ⭐⭐⭐⭐⭐ (High practical utility for commercial virtual try-on, avatar creation, and general region-controllable portrait editing)