Layering Virtual Try-On¶

Conference: ECCV 2026
Paper: CVF Open Access
Area: Segmentation
Keywords: Virtual Try-On, Layering Try-On, Diffusion Transformer, Cross-Frame Synthesis, Mask-Free Editing

TL;DR¶

Addressing the core limitation where conventional virtual try-on models erase existing outfits via cloth-agnostic masks, this paper disentangles general VTON priors from specific layering logic through a two-stage mask-free training paradigm, achieving realistic clothing addition and de-layering while faithfully preserving inner-layer garments.

Background & Motivation¶

Virtual Try-On (VTON) has made rapid strides toward commercial applications in e-commerce and digital styling, yet prevailing paradigms remain strictly limited to single-garment replacement. In authentic fashion scenarios, apparel is inherently layered—a blazer exposes an inner collared shirt, and a cardigan unzips over a graphic tee. However, mainstream methods overwhelmingly rely on cloth-agnostic representations, applying human parsing masks to erase all existing garments on the target person. This design choice irreversibly destroys the structural context of the subject's outfit, making it structurally impossible to add an outer layer without wiping out the inner garments beneath. Furthermore, standardized benchmarks such as VITON-HD and DressCode exclusively curate isolated garment-swapping pairs, leaving models oblivious to occlusion dynamics and layering hierarchy.

While recent mask-free editing attempts seek to circumvent manual masking by directly generating edits via diffusion models, collecting high-quality, large-scale paired data of layered garments remains prohibitively expensive. Crucially, human subjects naturally shift posture and orientation when donning successive clothing layers, introducing substantial spatial misalignment between consecutive frames. Directly fine-tuning diffusion backbones on scarce, pose-variant layering pairs forces the network into a destructive trade-off: simultaneously learning complex layering logic, non-rigid deformation, and fine-grained texture fidelity from insufficient data. Consequently, the model might retain an inner shirt collar but suffers catastrophic degradation in garment identity, warping buttons and blurring sleeve textures.

To overcome this bottleneck, this work introduces a pivotal insight: the challenges of layering virtual try-on must be disentangled into two complementary competencies—general VTON priors (deformation, identity preservation, and shading) and specific layering knowledge (layering order and occlusion reasoning). Core idea: train a diffusion backbone on mask-free, pose-mismatched pairs automatically synthesized from unstructured fashion videos to establish robust deformation and placement priors, followed by highly sample-efficient fine-tuning on a compact layering dataset augmented via temporal reversal.

Method¶

Overall Architecture¶

The framework operates under a two-stage disentangled training paradigm. Given an input person image $I_M$, a reference garment $I_G$, a target human pose $I_P$, and a textual edit instruction $T$ (such as add a dark blue denim blouse or remove a gray sweater), the system synthesizes a photorealistic try-on result $I_T$ that composites or removes the designated clothing layer while keeping non-target inner layers intact.

The underlying model is built upon Qwen-Image-Edit, comprising a Variational AutoEncoder (VAE) as the image latent tokenizer, a Qwen2.5-VL vision-language encoder for multimodal conditioning, and a Multimodal Diffusion Transformer (MMDiT) backbone optimized via Low-Rank Adaptation (LoRA). Stage 1 synthesizes pose-mismatched, mask-free pairs from fashion videos to build universal deformation priors. Stage 2 freezes the general foundation and fine-tunes the network on a curated video dataset of sequential dressing events, utilizing temporal reversal to model both layering and de-layering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Fashion Video Sequences<br/>Consistent outfit / Diverse poses"] --> B["Cross-Frame Pose-Mismatched Prior Synthesis<br/>Segmentation + Inpainting + Inter-frame pairing"]
    B --> C["MMDiT Dual-Stream Backbone<br/>Qwen2.5-VL Semantics + VAE Visual Latents"]
    D["Real Sequential Dressing Videos<br/>Temporal outfit clusters"] --> E["Temporal Reversal Augmentation<br/>Forward Layering ↔ Reverse De-Layering"]
    C --> F["Two-Stage Disentangled Fine-Tuning<br/>Prior Saturation → Layering Adaptation"]
    E --> F
    F --> G["Layering Try-On Output<br/>Accurate occlusion + Inner layer preservation"]

Key Designs¶

1. Cross-Frame Pose-Mismatched Prior Synthesis: Unlocking Non-Rigid Spatial Alignment

To eliminate reliance on cloth-agnostic masks while teaching the network robust spatial warping, this design leverages unstructured online fashion videos featuring individuals wearing identical outfits across diverse poses. Using SAM 2 and whole-body pose estimators, original garments are segmented from video frame $F_i$. An off-the-shelf inpainting diffusion model guided by LLM captions replaces the original outfit with a novel garment, producing synthetic frame $F_i'$. Crucially, instead of using $(F_i', F_i)$ as a trivial static pair, the training instance pairs $F_i'$ with a temporally distant frame $F_j$ from the same sequence ($i \neq j$): $$(I_M, I_G, I_P, T, I_T) := (F_i', G_j, P_j, T_j, F_j)$$ Because $F_i'$ and $F_j$ exhibit significant natural pose discrepancies, the network cannot rely on trivial pixel-to-pixel identity mapping. Instead, it is compelled to learn genuine non-rigid spatial deformation, realistic light transport, and garment identity preservation directly from raw pixel context, constructing a strong mask-free prior that handles real-world spatial misalignment during subsequent layering.

2. Temporal Reversal Augmentation: Symmetric Supervision for Layering and De-Layering

Real-world layering sequences are labor-intensive to capture, yielding only 60 curated video sequences in the baseline collection. To maximize data efficiency and address both garment addition and removal, the method exploits physical time-reversal symmetry. Videos are segmented into discrete temporal clusters $\{C_1, C_2, \dots, C_K\}$ corresponding to stable outfit configurations, with vision-language models generating transitional prompts $T_{m \to m+1}$. The forward instance $(F_i, G_j, P_j, T_{m \to m+1}, F_j)$ supervises the addition of a garment (add <garment>). Inverting the temporal arrow yields a valid reverse transition $(F_j, G_i, P_i, T_{m+1 \to m}, F_i)$, where textual inversion rules automatically rewrite the prompt to remove <garment>. This formulation doubles the effective size of the scarce real-world layering dataset without manual annotation and equips the diffusion model with symmetric reasoning capabilities across both forward occlusion and un-occlusion.

3. Dual-Stream MMDiT Adaptation: Context-Aware Low-Rank Conditioning

The generation backbone employs Qwen-Image-Edit to balance semantic adherence and textural fidelity. The Qwen2.5-VL stream processes high-level textual instructions $T$ and reference garment semantics, while the VAE encodes the noised target latent, reference garment visual features, and pose map. The MMDiT architecture enables cross-attention communication between both streams across transformer blocks. LoRA adapters are injected into attention projection layers and trained using the standard denoising objective. By maintaining architectural and loss consistency across both training stages, the model retains full generative expressiveness while adapting smoothly from general spatial warping to nuanced multi-layer boundary composition.

Key Experimental Results¶

Main Results¶

On the proposed LVTON benchmark containing 532 real-world evaluation pairs, the model is benchmarked against leading mask-free try-on frameworks and multimodal foundation models.

Table 1: Quantitative comparisons on our LVTON dataset (ECCV 2026 Table 1)

Model	SSIM ↑	LPIPS ↓	FID ↓	KID×10³ ↓
OmniTry [16]	0.810	0.176	54.585	3.165
Any2AnyTryon [19]	0.787	0.229	89.567	18.645
Nano Banana [11]	0.798	0.195	49.179	2.362
HunyuanImage-3.0-Instruct [6]	0.767	0.246	88.070	15.561
Ours	0.843	0.127	48.957	1.463

Note: Absolute FID values are statistically inflated across all baselines due to the compact test set size (532 images).

Table 2: Traditional VTON Benchmarks (VITON-HD & DressCode, ECCV 2026 Table 2 excerpt)

Model Category / Method	VITON-HD SSIM ↑	VITON-HD LPIPS ↓	VITON-HD FID ↓	DressCode SSIM ↑	DressCode LPIPS ↓	DressCode FID ↓
IDM-VTON [9] (Cloth-Agnostic)	0.870	0.102	6.290	0.920	0.062	8.640
FitDit [25] (Cloth-Agnostic)	0.899	0.066	4.731	0.926	0.043	2.638
CATVTON [10] (Mask-Free)	0.870	0.057	5.425	0.892	0.046	3.992
OmniTry [16] (Mask-Free)	0.698	0.368	57.871	0.842	0.188	25.364
Stage 1-only (Zero-Shot)	0.887	0.086	5.235	0.898	0.101	4.710
Ours (Full Fine-tuned)	0.928	0.055	3.235	0.948	0.042	2.598

Ablation Study¶

The ablation investigates the indispensability of the two-stage training paradigm and the contribution of temporal reversal augmentation on the LVTON benchmark.

Table 4: Ablation study on training components on LVTON dataset (ECCV 2026 Table 4)

Config	SSIM ↑	LPIPS ↓	FID ↓	KID×10³ ↓	Note
Stage 2-only	0.817	0.192	58.561	13.242	Direct training on scarce layering data alone collapses deformation fidelity
Stage 1-only	0.821	0.183	56.663	12.286	Trained solely on garment swaps; fails to execute additive layering commands
Ours w/o Aug	0.832	0.131	49.353	3.003	Removing temporal reversal halves data and noticeably degrades scores
Ours (Full Model)	0.843	0.127	48.957	1.463	Two-stage disentangling combined with augmentation achieves peak metrics

Key Findings¶

The Pitfall of Direct End-to-End Training: Omitting Stage 1 pre-training (Stage 2-only) leads to a severe surge in KID to 13.242 (nearly $9\times$ worse than the full model), empirically demonstrating that scarce layering data cannot simultaneously supervise spatial deformation and compositional reasoning.
Generalization of Synthetic Priors: The Stage 1-only model achieves a zero-shot FID of 5.235 on VITON-HD without observing any target benchmark data, surpassing fully trained cloth-agnostic models like IDM-VTON (6.290) and proving the general transferability of cross-frame video learning.
Prior Saturation Dynamics: Varying the volume ratio of Stage 1 synthetic data relative to Stage 2 data reveals that performance gains plateau around a 4.0× data ratio, demonstrating high sample efficiency and bounding synthetic data requirements.

Highlights & Insights¶

Paradigm Shift in Task Formulation: By formalizing Layering Virtual Try-On (LVTON), the paper moves the community beyond the traditional cloth-agnostic assumption, enabling realistic multi-garment composition and de-layering for real-world styling.
Cross-Temporal Video Mining: Pairing temporally mismatched frames ($i \neq j$) from videos with consistent outfits provides a self-supervised solution to non-rigid spatial alignment, forcing the model to master deformation without synthetic warping artifacts.
Physical Symmetry as Data Augmentation: Utilizing the temporal reversibility of dressing and undressing sequences naturally doubles scarce layering pairs and cleanly unifies outer garment addition and removal into a single framework.

Limitations & Future Work¶

Upstream Quality Bottlenecks: Synthetic pairs in Stage 1 depend heavily on the precision of SAM 2 segmentation and diffusion inpainting models; boundary halo artifacts or unnatural inpainting artifacts can leak into learned VTON priors.
Tail Dressing Geometries: While the system exhibits robust in-the-wild generalization, extreme multi-layer dressing interactions (e.g., rigid puffy parkas compressing oversized heavy-knit turtlenecks) still occasionally suffer from unnatural geometric overlap.
Non-Symmetric Fabric Dynamics: Temporal reversal assumes dressing and undressing are mechanically symmetric, omitting physical non-reversibilities such as fabric friction pulling inner hems or hair dishevelment during overhead de-layering.

vs. OmniTry [16] & Any2AnyTryon [19]: While these approaches operate mask-free, their training regimes are anchored to single-item replacement, causing them to overwrite existing inner layers upon receiving layering instructions. The proposed method explicitly decouples layering logic to retain non-target garments.
vs. IDM-VTON [9] & FitDit [25]: Conventional cloth-agnostic methods require explicit segmentation masks that erase underlying clothes, completely precluding layered try-on. The proposed framework removes this mask constraint entirely while outperforming these baselines on standard single-item benchmarks.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates the layering virtual try-on challenge and resolves it with an elegant two-stage disentangled learning pipeline.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across the dedicated LVTON benchmark, user studies, cross-domain VITON-HD/DressCode testing, and detailed ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, disciplined terminology, and rigorous analytical argumentation.
Value: ⭐⭐⭐⭐⭐ Breaks through the decadelong single-garment replacement constraint, offering immense practical utility for e-commerce and interactive styling systems.