H-Adapter: Pose-Robust Hairstyle Transfer via Attention-Derived, Source-Aligned Hair Masks¶

Conference: ECCV 2026
arXiv: 2606.25578
Code: None
Area: Image Generation / Diffusion Models
Keywords: Hairstyle Transfer, Diffusion Inpainting, Cross-Attention, Region-Specific Loss, Robust Pose

TL;DR¶

H-Adapter fine-tunes IP-Adapter with a "region-specific loss" such that the adapter injects the reference hairstyle only in the hair region while keeping non-hair regions unchanged. After training, the cross-attention naturally separates hair from non-hair regions, allowing the derivation of a coarse, source-pose-aligned mask to guide a two-stage diffusion inpainting process. This enables accurate hairstyle placement and alignment with the face geometry even when there is a significant pose discrepancy between the source and reference images.

Background & Motivation¶

The goal of hairstyle transfer is to transplant the hairstyle shape & color of a reference image to a source image while preserving the source identity, background, and clothing. Applications such as virtual hair try-on make this a valuable and practical image editing task. The key challenge lies in its "region-dependent" nature: the model must reflect the reference hairstyle features in the hair region, while keeping the non-hair regions completely intact. These two objectives are naturally in conflict; without careful balance, edits will leak and modify the face or background. More critically, hair is non-rigid and depends heavily on head pose. A turn of the head dramatically changes the spatial position and outline of the hair. Thus, when there is a large pose discrepancy between the source and reference, the model must first resolve "where the hair should exist on the source head geometry and what its approximate shape should be," which is precisely where existing methods fail most frequently.

Historically, GAN-based methods (such as Barbershop, Style-Your-Hair, and HairFastGAN) were dominant. They generally rely on aligned face datasets and operate well in controlled environments, but their robustness is limited on unconstrained images. Subsequently, diffusion-based approaches (e.g., Stable-Hair, HairFusion) emerged to improve generalization, but still suffer from two common failures under pose mismatch: first, inaccurate localization of the hair region, causing misplaced or over-expanded editing; second, even if the global placement is reasonable, they struggle to replicate the fine-grained structure and texture of the reference. These methods either rely on explicit alignment, force pose clues (like DensePose) into the attention layer, or perform latent code manipulation, all of which essentially introduce external pose priors to correct alignment. The authors observe a simpler path: image-conditioned adapters like IP-Adapter already inject reference features via cross-attention. If the attention can naturally learn "where is hair and where is not," the attention map can directly serve as a spatial guide aligned with the source pose, eliminating the need for external pose priors.

However, a standard IP-Adapter is trained for "global" condition injection. Using it directly for hairstyle transfer splatters the reference features across the entire face, and the attention cannot segregate hair and non-hair. The Core Idea is to re-train IP-Adapter with a region-specific loss—performing standard diffusion denoising in the hair region, while regularizing the non-hair region to match the predictions of the original diffusion model, forcing the adapter to take effect only in hair regions. The resulting cross-attention naturally forms a clear spatial separation between the hair and non-hair regions. A coarse, source-aligned mask can then be binarized from this attention to constrain a two-stage inpainting process, placing the hairstyle to fit the source head geometry.

Method¶

Overall Architecture¶

Given a source image \(x_s\) and reference image \(x_r\), the goal is to transfer the reference hairstyle to the source image while preserving non-hair content. The method consists of three components: first, training IP-Adapter with a region-specific loss to become the "hair-only" H-Adapter (training phase); second, during inference, using a source-aligned coarse attention mask to restrict diffusion inpainting to target hair regions via a "warm-up \(\rightarrow\) stage 1 \(\rightarrow\) stage 2" coarse-to-fine pipeline; third, leveraging the plug-and-play capability of the adapter to enable extensions like text-to-image, hair color control, and identity preservation.

Training is performed on a standard text-to-image backbone (SD v1.5), while the inpainting variant is used during inference. Training data consists of triplets \((x_i, x_j, M_i)\), where \(x_i\) is the target image, \(x_j\) is the conditional image for IP-Adapter, and \(M_i\) is the binary hair mask extracted from \(x_i\); FFHQ uses self-pairing (\(x_j=x_i\), where the same image acts as both reference and target), and CelebV-HQ uses two frames of the same identity (\(x_i\ne x_j\), with different poses/expressions). During inference, an instruction-based editing model (FLUX.2) is first used to convert the source image into a "bald base image" \(\tilde{x}_s\) without hair, on which the inpainting is subsequently performed.

The diagram below describes the coarse-to-fine pipeline during inference (for the training-side region-specific loss, see "Key Designs 1"):

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source x_s + Reference x_r"] --> B["Bald Base Image<br/>FLUX.2 removes source hair to get x̃_s"]
    B --> C["Warm-up 1 step<br/>Run one step with all-ones mask<br/>Extract cross-attention"]
    C --> D["Source-aligned Coarse Mask<br/>Sum and binarize cross-attentions<br/>excluding separator token to get M_attn"]
    D --> E["Stage 1 Inpaint<br/>Denoise with M_attn for ~20 steps<br/>to obtain intermediate y_mid"]
    E --> F["Pixel-level Mask<br/>Segmentation model predicts M_seg on y_mid"]
    F --> G["Stage 2 Inpaint<br/>Rerun from initial noise using M_seg<br/>to obtain final y_fine"]

Key Designs¶

1. Region-Specific Loss: Incorporating "Edit Hair Only, Do Not Touch Others" into the Training Objective

Standard IP-Adapter is trained for global injection. Utilizing it directly for hairstyle transfer contaminates the face and background with reference features, and the attention cannot segregate hair. The authors address this by splitting the denoising loss into two regional parts. In the hair region, standard diffusion denoising is computed, but only within the mask \(M\)—meaning the noise predictor with H-Adapter is trained to fit the ground-truth noise, so the reference hairstyle appearance is learned only at the hair region:

\[\mathcal{L}_{\mathrm{hair}}=\mathbb{E}_{z,\epsilon,t}\left[\left\|(\epsilon-\epsilon_{\theta_{\mathrm{H}}}(z_t,t,c_t,c_i))\cdot M\right\|_2^2\right]\]

Conversely, for the non-hair region, instead of fitting ground-truth noise, the output of H-Adapter is regularized to match the prediction of the "original diffusion model without the adapter" \(\epsilon_{\theta_0}\), instructing the model to remain unchanged in these regions:

\[\mathcal{L}_{\mathrm{non\text{-}hair}}=\mathbb{E}_{z,\epsilon,t}\left[\left\|(\epsilon_{\theta_{\mathrm{H}}}(z_t,t,c_t,c_i)-\epsilon_{\theta_0}(z_t,t,c_t))\cdot(1-M)\right\|_2^2\right]\]

The total loss is the sum of both terms, with the non-hair term weight \(\lambda_{\mathrm{non\text{-}hair}}\) fixed at \(0.1\) throughout. The elegance of this design lies in explicitly anchoring the non-hair region to the pretrained model rather than just supervising the hair region, changing "where can be edited vs where cannot" into two opposing objectives. Because the conditional branch is forced to be responsible only for the hair, its cross-attention naturally pushes spatial distances between hair and non-hair regions—this separation serves as the basis for deriving the mask later.

2. Source-Aligned Coarse Attention Mask: Free Extraction of a "Where is Hair" Spatial Map from Attention

A useful byproduct of the region-specific loss training is that among the 16 IP-Adapter tokens, exactly one token (denoted as the separator token \(t_s\)) consistently attends only to the non-hair region, acting as a "separator", while the other tokens focus on the hair region. Thus, by summing the cross-attention maps of all tokens except the separator token and binarizing it, one obtains an attention-derived coarse inpainting mask \(M_{\mathrm{attn}}=\mathrm{Binarize}\left(\sum_{k\ne s}\mathrm{CA}(t_k)\right)\). In practice, with \(K=16\), the authors experimentally find that \(t_8\) is the separator token.

The authors validate why \(t_8\) is the separator and why it is stable in two layers. First, in a controlled self-paired setting with ground-truth hair masks, each of the 16 tokens is evaluated as a candidate separator token by comparing the IoU of the aggregated mask (excluding that token) and the token's own mask against the ground truth. \(t_8\) yields the highest average IoU (0.549) across 3,000 samples. Second, in terms of value vector norm, \(t_8\) inherently has the smallest norm in the pretrained IP-Adapter (contributing least to the reference appearance injection). The region-specific loss further suppresses its relative norm from approximately 0.22 to 0.15, while its attention map consistently aligns with non-hair regions across different random seeds—demonstrating this is not an accidental single-run event but a stable non-hair separator amplified by training. Regarding mask resolution, the authors compare 8×8, 16×16, and 32×32, selecting 16×16 because it provides the most suitable coarse localization for the overall hair area and achieves the highest average IoU.

3. Two-Stage Coarse-to-Fine Inpainting (with Warm-up Step): Coarse Mask Localization and Fine Mask Refining

The attention-derived \(M_{\mathrm{attn}}\) is only a coarse localization and insufficient for pixel-level editing. Thus, the authors design a "warm-up + two-stage" coarse-to-fine pipeline. The difficulty is that there is no source-aligned mask at the beginning of inference. The proposed solution is to first run a warm-up denoising step with an all-ones mask, solely to extract the cross-attention and derive \(M_{\mathrm{attn}}\). Once \(M_{\mathrm{attn}}\) is obtained, stage 1 performs ~20 denoising steps with it to yield an intermediate image \(y_{\mathrm{mid}}\). Although \(y_{\mathrm{mid}}\) may contain stray regions due to the coarse mask, it preserves the source pose and shape. In stage 2, a pretrained segmentation model (BiSeNet) predicts a pixel-level hair mask \(M_{\mathrm{seg}}\) on \(y_{\mathrm{mid}}\). Denoising is then rerun from the initial noise latent \(z_T\), using \(M_{\mathrm{seg}}\) as the final inpainting mask to produce the final output \(y_{\mathrm{fine}}\).

This "coarse-to-fine" strategy is critical for pose robustness. In contrast, the baseline IP-Adapter without region-specific loss suffers from diffused attention, yielding a binarized mask of nearly all ones and editing globally. Even if the \(t_2\) mask that stably highlights the face is used as an alternative, it leads to contour drift and propagates to a misaligned \(M_{\mathrm{seg}}\), causing boundary drift. Conversely, because H-Adapter provides source-aligned guidance in the first stage, the second stage can extract an accurate pixel-level mask, strictly restricting the edits to the hair region.

4. Time-Step Reference Gating and Plug-and-Play Extensions: One Adapter for Multiple Downstream Tasks

In addition to using the mask in the two-stage pipeline, the authors recompute a spatial mask \(M_t\) at each time step to gate the reference injection term in cross-attention. The standard cross-attention is computed normally, while the contribution of the reference branch is element-wise multiplied by \(M_t\) and added back with weight \(\lambda\). A smaller \(\lambda\) respects the text prompt more, while a larger \(\lambda\) remains loyal to the reference hairstyle:

\[Z=\mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d}}\right)V+\lambda\Bigl(M_t\odot\mathrm{softmax}\!\left(\frac{Q{K'}^{\top}}{\sqrt{d}}\right)V'\Bigr)\]

where \((K,V)\) come from the standard text context and \((K',V')\) from the reference condition of H-Adapter. This gating mechanism enables the "one-time training, plug-and-play" capability of H-Adapter: it can perform reference-guided text-to-image generation without changing source image conditions, handle auxiliary hair color control by adding a hair color prompt (e.g., "a photo of a person with blue hair"), or be combined with identity preservation adapters (e.g., IP-Adapter FaceID Plus). In the latter case, the H-Adapter branch is gated by the hair mask \(M_t\) while the FaceID branch is gated by the complementary mask \(1-M_t\), allowing both hairstyle transfer and identity preservation without retraining.

Loss & Training¶

The final objective is \(\mathcal{L}=\mathcal{L}_{\mathrm{hair}}+\lambda_{\mathrm{non\text{-}hair}}\mathcal{L}_{\mathrm{non\text{-}hair}}\) with \(\lambda_{\mathrm{non\text{-}hair}}=0.1\). H-Adapter is initialized from IP-Adapter-Plus weights and fine-tuned on SD v1.5, using CLIP-ViT-H-14 as the image encoder. During training, the text condition is fixed to the generic prompt "a photo of a person". The dataset, after cleaning, comprises 68,058 FFHQ images (self-paired) and 9,188 CelebV-HQ pairs (same identity, selecting frame pairs with the largest yaw discrepancy). Training is performed on a single RTX 5090 for 16,000 steps with a batch size of 8 and a learning rate of \(1\times10^{-4}\). Inference takes approximately 3.05 seconds per pair, which is slower than HairFastGAN (0.78s) but faster than Stable-Hair (8.82s) and HairFusion (66.28s).

Key Experimental Results¶

Main Results¶

The authors construct two evaluation subsets on CelebA-HQ, each containing 3,000 pairs: "pose-agnostic" (no pose filtering) and "pose-different" (source-reference yaw discrepancy \(>15^{\circ}\)). Metrics include FID / FID_CLIP (visual realism), SSIM / PSNR (preservation of non-hair regions, computed on the non-hair intersection), and CLIP-I (hairstyle transfer fidelity). The table below shows the main results on the pose-different subset:

Method	FID ↓	FID_CLIP ↓	SSIM ↑	PSNR ↑	CLIP-I ↑
Ours	12.47	3.98	0.831 (second-best)	23.06	0.659
IP-Adapter (t8)	15.27	8.83	0.803	21.56	0.639
IP-Adapter (t2)	12.53	4.26	0.825	22.70	0.651
FLUX.2	12.66	5.18	0.904	25.35	0.643
HairFusion	28.03	8.80	0.756	17.26	0.626
Stable-Hair	25.79	8.70	0.798	22.39	0.640
HairFastGAN	12.78	4.53	0.817	24.40	0.649
HairCLIPv2	13.44	7.91	0.824	23.63	0.623
Style-Your-Hair	15.95	8.54	0.816	22.80	0.649

Ours achieves the best performance in FID, FID_CLIP, and CLIP-I. FLUX.2 scores the highest SSIM/PSNR (indicating strong source preservation) but is inferior to Ours in FID, FID_CLIP, and CLIP-I. Ours achieves second-best SSIM and competitive PSNR, indicating a superior balance between "faithful hairstyle transfer" and "preserving non-hair content." The trend is consistent on the pose-agnostic subset, where Ours achieves the best FID, FID_CLIP, SSIM, and CLIP-I.

VLM-as-a-judge (using GPT-4o as the judge, with Krippendorff's \(\alpha \ge 0.90\) for consistency) rates entries from 1 to 5 along three axes: HFS (Hairstyle Fidelity Score), NPS (Non-hair Preservation Score), and AQS (Artifact Quality Score, higher means fewer visible seams or color bleeds):

Method	HFS ↑	NPS ↑	AQS ↑
Ours	3.11	4.23	3.73
HairFusion	2.55	3.55	3.09
Stable-Hair	2.90	3.42	2.75
HairFastGAN	2.83	3.91	3.29
HairCLIPv2	2.10	4.05	3.57
Style-Your-Hair	2.87	3.98	3.61

Ours ranks highest on all three axes and is the only method to exceed 3.0 in HFS (ratings are generally low due to strict criteria requiring simultaneous matching of color, texture, length, contour, and parting). Other baselines have distinct bottlenecks: HairCLIPv2 exhibits high NPS but the lowest HFS, while Stable-Hair shows the opposite. In a user study with 3,193 pairwise preference votes from 53 subjects, Ours is preferred overall by 72.7% against all baselines (the smallest gap is against HairFastGAN at 55.3%, and over 80% against HairCLIPv2/HairFusion).

Ablation Study¶

Configuration	Key Metrics (pose-different)	Description
Ours (FFHQ + CelebV-HQ)	SSIM 0.830 / PSNR 23.05	Complete training data
Ours w/o CelebV-HQ	SSIM 0.761 / PSNR 18.35	Removing cross-frame pairs leads to major drops in SSIM/PSNR
Ours (t8 mask, attention extraction)	See main table	Complete pipeline
w/o Region-Specific Loss (IP-Adapter t8)	FID 15.27 / FID_CLIP 8.83	Attention is diffused, yielding near-global masks
w/o Region-Specific Loss (IP-Adapter t2)	FID 12.53 / FID_CLIP 4.26	Changing to t2 mask still suffers from contour drift

Key Findings¶

The region-specific loss is fundamental to deriving the "source-aligned mask." Without it, the IP-Adapter baseline fails to obtain a reliable separator token at \(t_8\), leading to diffused attention and a binarized mask of nearly all ones (causing near-global editing). Even switching to a \(t_2\) mask leads to contour drift. The loss transforms attention from a "globally smeared" state to a localized, usable segmentation signal for hair and non-hair separation.
Including CelebV-HQ cross-frame pairs primarily improves SSIM/PSNR (removing them drops SSIM from 0.830 to 0.761, and PSNR from 23.05 to 18.35). Since the reference and target frames share the same identity but have different poses and expressions, it forces the model to learn stronger invariance in non-hair regions, mitigating unwanted changes. Conversely, CLIP-I is comparable under both data compositions, indicating self-pairing alone captures hairstyle properties well.
A mask resolution of 16×16 achieves the highest IoU with the ground-truth hair mask. If it is too coarse, localization is inaccurate; if too fine, it is easily dominated by high-frequency structures. For the separator token, \(t_8\) obtains the highest average IoU, and value vector norm analysis confirms its stability rather than random assignment.
In a controlled experiment where all methods use the same bald base image, Ours still outperforms all others on FID, FID_CLIP, and CLIP-I, proving that the advantage is not solely attributable to the FLUX-depilated input. In fact, most baselines perform worse on the bald base image because of distribution shifts.

Highlights & Insights¶

The most striking "aha" moment is directly converting the regional structure of the loss into the spatial structure of the attention. Without feeding any pose priors into the model, simply applying a denoising loss in the hair region and anchoring to the original model in the non-hair region forces the cross-attention to learn hair separation automatically. It is simple, and the mask is naturally aligned with the source pose.
The discovery of the "separator token" is highly elegant: exactly one of the 16 tokens stably attends only to non-hair regions, and excluding it from the aggregation yields the hair mask. Furthermore, the authors do not stop at describing the phenomenon but prove via value-norm analysis that the region-specific loss "amplifies an existing weak bias" rather than producing an accidental artifact, adding strong credibility.
Employing an all-ones mask for a "warm-up step" solely to extract attention, followed by the two-stage pipeline, elegantly resolves the chicken-and-egg problem where no mask exists at the start of inference. This trick is readily transferable to other editing tasks that require bootstrapping spatial masks from attention.
Time-step gating combined with complementary masking (gating the hair branch with \(M_t\) and the identity branch with \(1-M_t\)) provides a highly practical plug-and-play paradigm, enabling a single adapter to handle text-to-image, hair color control, and identity preservation without retraining.

Limitations & Future Work¶

The authors acknowledge validation is limited to the SD1.5 / IP-Adapter-Plus backbone. Different backbones and adapters might exhibit different token behaviors and attention patterns, which means the separator token selection and mask extraction strategies would need recalibration for new architectures.
The benchmark only covers faces without head occlusions. Scenarios involving hats, scarves, helmets, etc., are out of scope. Robustly distinguishing hair from non-hair head-coverings remains an open challenge.
The authors observe some failures: the model occasionally struggles to preserve specific spatial relationships between the reference hair and face structures, such as how bangs extend relative to the forehead or eyebrows. Fine-grained hair-face geometry modeling remains a bottleneck.
The pipeline relies on external components (FLUX.2 for de-hairing, BiSeNet for segmentation), resulting in a longer pipeline. Taking 3.05 seconds per pair is faster than diffusion baselines but slower than GAN encoder methods, and the quality of the base de-haired image affects downstream results.

vs HairFusion: Both extract spatial clues from cross-attention, but HairFusion injects pose clues (DensePose) into attention, performs latent blending in the last \(n\) steps, and combines masks with the source hair mask. H-Adapter relies on region-specific training to improve signal selectivity, generating a coarse mask tightly bound to the hair region, whereas HairFusion's masks tend to expand into neighboring visual contexts.
vs HairFastGAN: The latter is a GAN encoder method that introduces a Rotate Encoder to transform facial latent codes for pose-aware transfer. H-Adapter avoids latent code transformation, adopts a diffusion inpainting pipeline, and yields better quality metrics (FID/CLIP-I) at the cost of speed (3.05s vs 0.78s).
vs MasaCtrl / DiffEdit / InstDiffEdit: These methods derive editing masks from attention/noise discrepancies for general editing localization. H-Adapter specializes in hairstyle transfer under pose mismatch, extracting "source-aligned hair masks," where the mask separation ability stems from dedicated region-specific training rather than general-purpose attention manipulation.
vs Direct IP-Adapter: Standard IP-Adapter injects conditions globally, contaminating non-hair regions, and its attention cannot segregate hair. The proposed region-specific loss is the key step to transforming it from a "global adapter" to a "hair-specific adapter."

Rating¶

Novelty: ⭐⭐⭐⭐ The idea of "inducing spatial separation in attention via regional loss structures and bootstrapping source-aligned masks" is elegant and straightforward. The finding and validation of the separator token are convincing.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Quantitative + qualitative + three-judge VLM evaluation + human preferences + controlled de-hairing experiments + comprehensive ablations on mask resolution, separator token, and training data.
Writing Quality: ⭐⭐⭐⭐ Methodology and analysis are well-organized. The appendix provides solid validation for the separator token's stability. While the two-stage pipeline may take a moment to digest initially, it maps well to the figure.
Value: ⭐⭐⭐⭐ Virtual try-on represents a clear practical application, and the plug-and-play extensions are highly practical. However, being tied to the SD1.5 backbone and relying on external de-hairing/segmentation means generalizing to new backbones still requires effort.