FDM-MFVT: Few-step Sampling Diffusion Model for Mask-Free Virtual Try-On

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Generation
Keywords: Virtual Try-On, Diffusion Models, Few-Step Sampling, Mask-Free Try-On, Noise Optimization

TL;DR

Addressing the computational latency and error-prone pre-processing of diffusion-based virtual try-on, FDM-MFVT introduces an Outfit-aware Noise Optimization module (OANO) and an Instruction-driven Try-on module (IDT) that enables high-fidelity mask-free virtual try-on in merely 6 inference steps.

Background & Motivation

Image-based virtual try-on (IVTON) seeks to synthesize a realistic image of a target individual dressed in a specified garment, serving as a critical cornerstone for online e-commerce, digital retail, and interactive fashion modeling. Recent advances in diffusion models, such as StableVITON, OOTDiffusion, and IDM-VTON, have substantially elevated visual realism and fine-grained textile texture rendering by harnessing powerful generative image priors. However, conventional diffusion-based try-on systems remain heavily burdened by two fundamental challenges: first, iterative denoising requires dozens of sampling steps (typically 30 to 50 iterations), incurring severe latency that hampers real-time user experiences; second, the synthesis pipeline critically relies on hand-crafted or auto-segmented agnostic masks, supplemented by auxiliary encoders for DensePose and human parsing maps. This complex multi-condition dependency makes inference fragile and computationally bloated—even minor mask misalignments inevitably cascade into distorted garment contours, erased hair strands, severed wrists, or residual ghosts of original clothes.

The core tension underlying these issues stems from the acute sensitivity of diffusion models to initial noise distributions during few-step sampling regimes, where abruptly cutting sampling steps causes severe structural collapse and blurred textures. Simultaneously, the lack of large-scale, high-quality paired datasets without mask annotations has hindered the emergence of robust mask-free architectures. Without explicit binary masks to partition replacement zones from retained human body parts, models struggle to discern clothing boundaries and preserve person identities.

The key insight of this paper is to investigate how initial noise priors influence sampling convergence—revealing that the singular vectors of inversion noise carry rich structural clues that permit rapid convergence to target data manifolds within very few steps. Core idea: decouple virtual try-on into outfit-aware noise optimization and lightweight instruction-driven diffusion adaptation, utilizing OANO to align Gaussian noise with garment-person semantics in the SVD singular subspace for 6-step convergence, and employing IDT to achieve mask-free multimodal conditioning without auxiliary pre-processors.

Method

Overall Architecture

FDM-MFVT is built upon a large-scale Diffusion Transformer backbone (Flux1.Fill). The framework operates end-to-end without requiring human parsing maps or explicit segmentation masks. Given a target garment image \(I_g\) and a person image \(I_p\), the Outfit-aware Noise Optimization (OANO) module first extracts joint garment-person representations and refines the initial isotropic Gaussian noise \(x_T\) via singular value decomposition (SVD) and cross-attention perturbation, producing an optimized starting noise state \(\hat{x}'_T\). Subsequently, the Instruction-driven Try-on (IDT) module maps visual and textual tokens into a shared contextual embedding sequence, feeding it alongside the optimized noise into a frozen DiT adapted with dual-branch LoRA layers. In just 6 sampling steps, the model denoises and decodes the latent representations into a \(1024 \times 768\) photorealistic try-on result.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Inp["Input: Person Image Ip + Garment Image Ig"] --> OANO["Outfit-aware Noise Optimization (OANO)<br/>SVD decomposition + cross-modal attention perturbation"]
    OANO --> NoiseOpt["Optimized Noise x'T<br/>Informative starting point for 6-step sampling"]
    Inp --> IDT["Instruction-driven Try-on (IDT)<br/>VAE visual latents concatenated with T5 text prompt"]
    NoiseOpt --> DiT["Flux1.Fill Inpainting DiT<br/>Dual-branch multimodal LoRA adaptation"]
    IDT --> DiT
    DiT --> Out["Output: High-fidelity mask-free try-on image (1024x768)"]

Key Designs

1. Outfit-aware Noise Optimization (OANO): Reconstructing Few-Step Initial Distributions

Standard Gaussian white noise carries no structural or semantic correlation with the target garment, forcing diffusion models to execute numerous iterations before settling into the correct data manifold. OANO addresses this by transforming random noise into a structured, outfit-aware starting point. The authors curate a noise-pair dataset \(\mathcal{D}_{\text{noise}} = \{(x_T, x'_T \mid Q(x'_T) > Q(x_T))\}\) via DDIM inversion and a perceptual quality filter \(Q(\cdot)\). Recognizing from singular vector analysis that \(x_T\) and \(x'_T\) share strong subspace similarities, OANO encodes the concatenated garment and person image into feature \(I_f = L(E(I_{gp}))\), while computing the SVD of random noise \(x_T = U \Sigma V^T\). A learnable cross-attention mechanism perturbs the noise components using outfit guidance: $\(Z_T = \mathrm{softmax}\left(\frac{Q_R K_T^T}{\sqrt{D_K}}\right) V_T\)$ where query \(Q_R\) is derived from image feature \(I_f\), and key-value pairs \(K_T, V_T\) are obtained from self-attended noise features. The perturbed representation is combined through a feed-forward network and reconstructed via inverse SVD into optimized noise \(\hat{x}'_T = U \times L(\mathrm{SA}(x'_{T_f})) \times V^T\). Operating with minimal parameter overhead in only 0.28 seconds, OANO allows a 6-step sampling trajectory to match or surpass traditional 30-step generation quality.

2. Instruction-driven Try-on (IDT): Eliminating Explicit Masks and Parsing Guidance

Existing try-on frameworks are acutely vulnerable to segmentation flaws in off-the-shelf parsing models. IDT reconceptualizes virtual try-on as an instruction-guided inpainting and completion task. Person and garment images are transformed into visual tokens \(z_i\) via a VAE encoder, while the natural language prompt \(P_t\) is encoded into textual tokens \(z_t\) via a frozen T5 encoder. These tokens are concatenated along the sequence length to form a unified prompt sequence \(Z = \mathrm{Concat}(z_i, z_t)\). The large Diffusion Transformer intrinsically resolves correspondence and boundary delineation across modalities, entirely removing the necessity for hand-drawn masks, DensePose UV coordinates, or body keypoint detectors.

3. Dual-Branch LoRA Adaptation: Reconciling Foundation Priors with Fashion Semantics

Full fine-tuning of massive DiT models risks catastrophic forgetting and excessive GPU resource demands. IDT incorporates an efficient low-rank adaptation structure. The normalized contextual sequence \(Z_n\) is processed through two parallel pathways: a multimodal attention branch (MM-Attn) capturing non-local cross-attribute relationships and a multi-layer perceptron (MLP) branch modeling dense local feature transformations: $\(Z_{\text{attn}} = \mathrm{MM\text{-}Attn}(Z_n), \quad Z_{\text{mlp}} = \mathrm{MLP}(Z_n)\)$ Their concatenated projections \(Z_{\text{proj}}\) are integrated with the input tokens via element-wise multiplication and residual addition: $\(Z_{\text{out}} = Z_{\text{proj}} \odot Z_n + Z_n\)$ By keeping the underlying Flux1.Fill backbone frozen and updating only the lightweight LoRA layers, the network effectively aligns complex garment textures with body postures while preserving pristine skin rendering and background context.

4. The MFVT Benchmark: 30,000 High-Resolution Mask-Free Pairs

To establish a benchmark for mask-free research, the authors curated the MFVT dataset. Leveraging high-performance models to generate wearing imagery from randomly sampled combinations in VTON-HD and DressCode, they eliminated all auxiliary mask layers, yielding triplets of original person, target clothing, and final dressed results. MFVT encompasses 30,000 standardized \(1024 \times 768\) pairs categorized into 10,671 upper-body items, 9,761 lower-body items, and 9,568 dresses, partitioned into 70% training and 30% testing subsets.

Loss & Training

The architecture employs a decoupled two-stage training scheme: - OANO Optimization: Trained on noise dataset \(\mathcal{D}_{\text{noise}}\) via mean squared error (MSE) to match high-quality inversion targets \(x'_T\): $\(\mathcal{L}_{\mathrm{OANO}} = \frac{1}{D} \|x'_T - \hat{x}'_T\|^2\)$ Trained on a single 80GB NVIDIA A800 GPU for 150 epochs with a batch size of 64. - IDT LoRA Adaptation: Trained on MFVT by optimizing the diffusion objective over noisy latents \(x_t\) conditioned on \(Z_{\mathrm{out}}\): $\(\mathcal{L}_{\mathrm{diff}} = \mathbb{E}_{x_0, t, \epsilon} \left[ \|\epsilon - \hat{\epsilon}_\theta(x_t, t, Z_{\mathrm{out}})\|^2 \right]\)$ Trained on two 80GB NVIDIA A800 GPUs for 15,000 steps with a batch size of 16.

Key Experimental Results

Main Results

The model was rigorously benchmarked under both mask-free settings (on MFVT against Any2AnyTryon) and traditional mask-based settings (on VTON-HD and DressCode against 8 state-of-the-art baselines).

Table 1: Quantitative evaluation on the MFVT dataset against Any2AnyTryon across 6 and 30 inference steps

Method	Category	LPIPS (6 steps) ↓	SSIM (6 steps) ↑	FID (6 steps) ↓	KID (6 steps) ↓	LPIPS (30 steps) ↓	SSIM (30 steps) ↑	FID (30 steps) ↓	KID (30 steps) ↓
Any2AnyTryon	Upper-body	0.324	0.769	11.247	1.421	0.147	0.854	8.471	0.725
FDM-MFVT (Ours)	Upper-body	0.087	0.887	8.341	0.725	0.085	0.893	8.341	0.712
Any2AnyTryon	Lower-body	0.475	0.774	11.452	1.213	0.102	0.867	8.527	1.054
FDM-MFVT (Ours)	Lower-body	0.083	0.879	8.316	0.968	0.082	0.880	8.219	0.945
Any2AnyTryon	Dresses	0.348	0.771	10.461	1.411	0.164	0.829	8.527	0.947
FDM-MFVT (Ours)	Dresses	0.102	0.872	8.431	0.901	0.091	0.883	8.286	0.872

Table 2: Comparison against 8 mask-based SOTA methods on unpaired VTON-HD and DressCode Upper-body test sets

Method	VTON-HD (6 steps) FID ↓	VTON-HD (6 steps) KID ↓	VTON-HD (30 steps) FID ↓	VTON-HD (30 steps) KID ↓	DressCode (6 steps) FID ↓	DressCode (6 steps) KID ↓	DressCode (30 steps) FID ↓	DressCode (30 steps) KID ↓
StableVTON	13.427	3.427	9.717	1.450	22.847	6.547	13.666	3.857
OOTDiffusion	16.362	5.871	12.268	3.214	37.106	7.839	29.383	4.302
CATVTON	13.949	3.946	10.263	2.370	23.185	3.147	12.720	1.910
TPD	16.235	6.692	13.124	3.527	25.073	2.978	12.957	1.874
IDM-VTON	12.539	2.849	9.261	1.253	21.849	3.041	11.879	1.813
ITA-MDT	11.054	3.014	8.889	1.397	20.437	3.256	11.928	1.871
OmniVTON	13.296	3.781	9.732	1.193	22.348	3.239	12.324	1.869
FastFit	12.316	3.258	8.731	1.264	21.316	3.124	11.542	1.824
FDM-MFVT (Ours)	8.687	1.201	8.659	1.010	11.231	1.401	10.963	1.394

Ablation Study

Table 3: Ablation evaluation of individual modules on the MFVT dataset

Config	LPIPS ↓	SSIM ↑	FID ↓	KID ↓	Note
w/o OANO	0.121	0.864	8.417	1.019	Standard Gaussian noise init; degradation in few-step perceptual fidelity
w/o IDT	0.135	0.859	8.378	1.142	Disabling instruction-driven module hurts cross-modal alignment (+48.4% LPIPS)
FDM-MFVT (Full Model)	0.091	0.879	8.363	0.865	Optimal synergy yielding sharp textures and accurate structural fit

Key Findings

• Few-Step Generational Leap: At only 6 inference steps, FDM-MFVT significantly exceeds the 30-step synthesis performance of Any2AnyTryon across all apparel types. In the Dresses category, Any2AnyTryon requires 30 steps to obtain 0.829 SSIM and 8.527 FID, whereas FDM-MFVT achieves 0.872 SSIM and 8.431 FID in 6 steps.
• Outperforming Mask-Guided Baselines Without Masks: Despite relying on zero explicit parsing masks, FDM-MFVT at 6 steps scores an FID of 11.231 on DressCode Upper-body, surpassing the 30-step performances of dedicated mask-based methods including FastFit (11.542) and IDM-VTON (11.879).
• Sub-Second Noise Conditioning: The OANO module incurs only 0.28s execution latency, effectively compressing the required sampling budget by an 80% reduction.

Highlights & Insights

• SVD Subspace Noise Conditioning: Leveraging the shared singular vector characteristics between random noise and inversion trajectories allows direct conditioning during initialization, overturning the assumption that diffusion models must start from unbiased isotropic noise.
• Robustness Against Segmentation Artifacts: Completely bypassing human parsing maps and agnostic masks eliminates wrist dismemberment, erroneous hair deletion, and clothing ghosting caused by mask boundary errors.
• MFVT Open Benchmark: Releasing 30,000 paired \(1024 \times 768\) high-resolution triplets across upper, lower, and dress categories establishes a solid benchmark for future mask-free virtual try-on research.

Limitations & Future Work

• Extreme Poses and Heavy Self-Occlusion: While demonstrating strong in-the-wild generalization on StreetVTON, severe limb occlusion (e.g., crossed arms concealing the chest) can occasionally yield subtle texture misalignments without explicit skeleton guidance.
• Multi-Layer / Layered Virtual Try-On: The framework currently targets individual garment replacement (single top, bottom, or dress), without explicitly modeling layered outerwear over inner shirts.
• Future Directions: Exploring implicit 3D human pose priors within the noise optimization space, and expanding toward video-based temporally consistent virtual try-on.

Related Work & Insights

• vs Any2AnyTryon: Any2AnyTryon pioneers mask-free try-on but suffers from severe blurring and dark pant ghosting under 6-step sampling (FID 11.452); FDM-MFVT drops 6-step FID to 8.316 through SVD-guided noise optimization.
• vs FastFit / ITA-MDT: FastFit and ITA-MDT achieve strong acceleration on masked try-on but remain bottlenecked by preprocessing pipelines; FDM-MFVT removes masks altogether while achieving better 6-step metrics than their 30-step outputs.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Innovative use of SVD singular vector properties to condition initial noise for few-step try-on)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated on MFVT, VTON-HD, DressCode, and in-the-wild StreetVTON with comprehensive ablations)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clear problem definition, mathematically coherent formulation, and well-structured empirical validation)
• Value: ⭐⭐⭐⭐⭐ (Crucial milestone toward latency-free, pre-processing-free commercial virtual try-on deployment)