Rethinking Garment Conditioning in Diffusion-based Virtual Try-On: Decouple, Don't Denoise¶

Conference: ECCV 2026
arXiv: 2511.18775
Code: To be confirmed (no public code link found)
Area: Image Generation (Note: originally specified as image_restoration, but this paper actually belongs to virtual try-on/conditional image generation; recommended to move to image_generation)
Keywords: Virtual Try-On, Diffusion Models, Spatial Concatenation, Conditional Decoupling, Full-Parameter Fine-Tuning, DeCo-VTON

TL;DR¶

DeCo-VTON reveals for the first time, through visualization of the noise prediction behavior of dual-UNet reference networks, the root cause behind the failure of full-parameter fine-tuning in spatial-concatenation-based virtual try-on: three functional conflicts caused by the coupling of garment condition and the denoising process. Correspondingly, three decoupling design principles are proposed (garment-free guidance, decoupled loss, and clean latent anchoring). Without modifying the network architecture, DeCo-VTON achieves or surpasses the try-on quality of the 1.8B-parameter state-of-the-art dual-UNet model Leffa, using only a single 860M-parameter UNet.

Background & Motivation¶

Image-based virtual try-on (VTON) aims to naturally "drape" a target garment onto a given person image, serving as a key technology in e-commerce and fashion. Current diffusion models have replaced GANs as the mainstream paradigm for VTON. Among them, dual-UNet architectures achieve state-of-the-art (SOTA) garment fidelity by employing a dedicated reference UNet to process garment images and inject multi-scale features into the main denoising UNet. Leffa, the current SOTA among dual-UNet methods, achieves optimal fidelity by feeding the reference network with the same timestep as the main network to obtain timestep-aligned garment features (where high \(t\) provides coarse structures and low \(t\) provides fine textures). However, dual-UNet designs double the parameters and computation—Leffa reaches 1.8B parameters, leading to substantial inference and GPU memory overhead.

Single-network architectures bypass the reference UNet, with spatial concatenation being the most straightforward approach: concatenating the latents of the garment and person images spatially to denoise them as a whole, which is simple and lightweight. However, a puzzling phenomenon occurs: both CatVTON (860M UNet) and Voost (11.9B DiT) report that full-parameter fine-tuning performs worse than attention-only fine-tuning, or even degrades performance. The consistent conclusion from two models differing by an order of magnitude in scale and having completely different architectures implies that the issue lies not with the backbones themselves, but with spatial concatenation as a condition injection method. This begs the question: why does full-parameter fine-tuning fail under spatial concatenation, and can it be resolved?

By visualizing the noise prediction behavior of the dual-UNet reference network for the first time—specifically, decoding its outputs at different timesteps into images via the VAE—this work uncovers a key insight: successful garment conditioning must be structurally decoupled from the denoising process. Dual-UNet naturally satisfies this condition: the reference network always processes clean garment latents, free from denoising target constraints. In contrast, spatial concatenation embeds garment latents within the denoising target, losing this structural decoupling and causing three functional conflicts: the CFG unconditional branch suppresses rather than enhances garment details due to residual garment information (Conflict 1); the training objective simultaneously demands garment reconstruction and conditional transfer, resulting in gradient competition (Conflict 2); and during inference, the garment latents are updated by the model's prediction chain, deviating from the forward diffusion timeline (Conflict 3). Core Idea: To identify for the first time the three fundamental conflicts causing the failure of full-parameter fine-tuning in spatial concatenation, and correspondingly propose three decoupling design principles (garment-free guidance, decoupled loss, and clean latent anchoring) to unleash the potential of full-parameter fine-tuning without altering the network architecture.

Method¶

Overall Architecture¶

DeCo-VTON uses CatVTON's architecture as the baseline (SD1.5 Inpainting UNet, 860M) without any network structure modifications. The input consists of a masked person image and a garment image, which are encoded by the VAE and concatenated along the height dimension to form the denoising input. All innovations of this model lie in a training and inference recipe with three design principles addressing the three conflicts mentioned above. The flowchart below illustrates the inference pipeline and shows where the three principles are integrated:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Person Image"] --> B["VAE Encoder"]
    C["Garment Image"] --> B
    B --> D["Spatial Concat<br/>(Person on Top, Garment on Bottom)"]

    D --> E["Denoising UNet<br/>(SD1.5 Inpainting, Full FT)"]

    F["Clean Latent Anchoring<br/>Garment latents provided by forward diffusion<br/>instead of being updated by prediction chain"] -.->|Inference| E
    G["Garment-Free Guidance<br/>CFG Unconditional Branch<br/>completely removes garment info"] -.->|Inference CFG| E

    E --> H["VAE Decoder"]
    H --> I["Try-On Output"]

    J["Decoupled Loss<br/>Loss computed only on person region<br/>Garment serves as condition without reconstruction"] -.->|Training| E

Key Designs¶

1. Garment-Free Guidance (GFG): Eliminating Garment Leakage in CFG Unconditional Branch

Standard CFG enhances the conditional signal by amplifying the difference between conditional and unconditional predictions. In CatVTON's spatial concatenation, although the unconditional input removes the clean garment latent from the condition channel, it still retains the noise-corrupted garment latent as it is part of the denoising target and cannot be directly discarded. Consequently, the unconditional branch still "sees" the garment information. The CFG difference signal no longer purely reflects the contribution of the garment condition; increasing the guidance scale actually suppresses garment details. This is the most severe of the three conflicts. The solution of DeCo-VTON is straightforward: completely remove all garment-related latents in the unconditional input (filling the garment region with zeros), yielding a truly garment-free unconditional branch:

\[ \mathbf{X}^{\text{gf}}_t = \text{Concat}_{\text{ch}}\left(\mathbf{z}^p_t \oplus \mathbf{0},\; \mathbf{M} \oplus \mathbf{0},\; \mathbf{z}^p_0 \oplus \mathbf{0}\right) \]

Now, the CFG difference signal \(\epsilon_\theta(\mathbf{X}_t, t) - \epsilon_\theta(\mathbf{X}^{\text{gf}}_t, t)\) correctly isolates the contribution of the garment condition. Increasing the guidance scale effectively enhances rather than degrades garment details. During training, the garment condition is randomly dropped with a 10% probability to teach the model to handle such garment-free predictions. Ablation studies show that GFG contributes the largest single-step gain (FID drops by 0.712, KID drops by 0.328), validating that Conflict 1 is indeed the most critical bottleneck.

2. Decoupled Loss (DL): Removing Gradient Interference from Garment Reconstruction

The standard training objective computes the noise prediction loss over the entire concatenated area (person + garment). This means the network receives two competing gradients: one demanding accurate reconstruction of the garment region, and the other demanding the transfer of garment information to the person region. These two objectives are misaligned—features optimized for pixel-level garment reconstruction are not necessarily optimal for conditioning person generation. In contrast, the reference UNet in dual-UNet architectures only extracts features without experiencing reconstruction loss, thus avoiding this problem. DeCo-VTON's solution exploits the natural division of spatial concatenation: since the person and garment are concatenated vertically (person on top, garment on bottom), the network's output naturally segregates into person and garment components. The decoupled loss computes the DREAM objective only on the person component—applying DREAM loss with target and input correction specifically to the person region, while the garment region is kept at fixed forward diffusion values without participating in gradient propagation. Thus, the garment region within the network constantly serves solely as a conditioning context, free from reconstruction gradient competition. Ablation experiments show that while DL offers moderate improvements at 512×384 resolution, removing DL at 1024×768 high resolution leads to a 0.313 increase in Unpaired FID, demonstrating that higher resolutions intensify gradient competition.

3. Clean Latent Anchoring (CLA): Aligning Garment Region Noise Schedule during Inference

A key to the success of the reference UNet in Leffa is that it receives the same timestep \(t\) as the main network, thereby providing timestep-aligned garment features—outputting coarse structural features at high \(t\) and transitioning to fine texture features at low \(t\). However, in standard inference with spatial concatenation, the garment latents are updated step-by-step by the model's own prediction chain, and their noise levels gradually deviate from the forward diffusion expectations: at high timesteps, the garment region is actually too clean, whereas at low timesteps, it becomes "over-denoised" by the model. Timestep alignment is completely lost. DeCo-VTON's idea is to use the known clean original garment latents to fix this issue: before each inference step, the correct noisy version corresponding to that timestep is computed directly from the clean garment latent via the forward diffusion formula, replacing the garment latent in the model's prediction chain:

\[ \bar{\mathbf{z}}^g_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}^g_0 + \sqrt{1 - \bar{\alpha}_t}\,\boldsymbol{\epsilon}_{\text{init}} \]

Here, \(\boldsymbol{\epsilon}_{\text{init}}\) is a fixed random noise that remains constant throughout the diffusion process to ensure consistency. This guarantees two things: zero error accumulation (as the garment latent is always derived from the clean original image rather than the prediction chain) and exact timestep alignment (as the noise level of the garment region precisely matches the network's expectation at that timestep). CLA is an inference-only modification—during training, forward diffusion already naturally provides timestep-aligned noisy garments, so no additional processing is required. In ablation studies, CLA yields the most significant improvement in KID (from 0.068 to 0.010), showing that alignment during the inference phase is critical for the generation quality.

Loss & Training¶

The entire network is trained using full-parameter fine-tuning with a learning rate of \(1 \times 10^{-5}\), a global batch size of 128, the AdamW optimizer, and gradient clipping at 1.0. The loss function is the decoupled DREAM objective (\(\lambda=10\)), applied only to the person region of the concatenated input. During training, the garment condition is randomly dropped with a 10% probability, using mixed-precision BF16. It is trained for 16K steps on VITON-HD and 32K steps on DressCode, taking approximately 10 hours and 20 hours, respectively, on 2× H200 GPUs. During inference, the DDIM scheduler is used with a guidance scale \(\omega=2.5\).

Key Experimental Results¶

Main Results¶

Table 1: Quantitative Comparison on VITON-HD

Method	Architecture	Paired FID↓	Paired KID↓	Paired LPIPS↓	Unpaired FID↓	Parameters
CatVTON	Single-UNet	5.425	0.411	0.057	9.015	860M
Leffa	Dual-UNet	4.540	0.050	0.048	8.520	1.8B
DeCo-VTON	Single-UNet	4.438	0.010	0.047	8.266	860M

Table 2: Quantitative Comparison on DressCode

Method	Architecture	Paired FID↓	Paired KID↓	Unpaired FID↓	Parameters
CatVTON	Single-UNet	3.992	0.818	6.137	860M
Leffa	Dual-UNet	2.060	0.070	4.480	1.8B
DeCo-VTON	Single-UNet	2.175	0.062	4.310	860M

On both datasets, DeCo-VTON reaches or surpasses the performance of Leffa (1.8B dual-UNet) with its 860M single-network architecture. In terms of efficiency, DeCo-VTON's inference speed is 1.3s (vs. Leffa's 2.7s), and peak GPU memory is 2.26GB (vs. Leffa's 3.91GB), which translates to 2.1× faster speed and 42% lower memory consumption compared to Leffa. In high-resolution (\(1024 \times 768\)) comparisons, DeCo-VTON similarly outperforms the 11.9B Voost on Paired FID (4.741 vs. 5.269) using only an 860M single network.

Ablation Study¶

Table 3: Progressive Ablation Study (VITON-HD)

Configuration	Paired FID↓	Paired KID↓	Conflict Addressed	Description
CatVTON (Attn-only)	5.425	0.411	—	Baseline, attention-only training
CatVTON (Full FT)	5.250	0.402	—	Minimal gain, verifying the three conflicts
+ GFG	4.538	0.074	Conflict 1	Largest single-step gain, confirming CFG leakage as the most severe bottleneck
+ DL	4.517	0.068	Conflict 2	Further improvement, more pronounced at high resolutions
+ CLA (=DeCo-VTON)	4.438	0.010	Conflict 3	Largest KID reduction, complete recipe

Key Findings¶

Garment-Free Guidance (GFG) contributes the largest single-step improvement (FID drops by 0.712, KID drops by 0.328), proving that garment leakage in CFG is the most severe bottleneck in spatial concatenation. CatVTON's original full-parameter fine-tuning (FID 5.250) only marginally improved upon attention-only training (FID 5.425) by 0.175, but with the integration of GFG, it instantly dropped to 4.538—indicating that the failure of previous full-parameter fine-tuning attempts was not because full-FT itself is ineffective, but because Conflict 1 remained unresolved.
Clean Latent Anchoring (CLA) leads to the largest drop in KID (from 0.068 to 0.010), demonstrating that timestep alignment during the inference phase is crucial to the realism of the generated distribution. This aligns with Leffa's claim that a timestep-aligned reference network is key to success.
The decoupled loss brings a relatively modest gain at \(512 \times 384\), but removing DL at \(1024 \times 768\) degrades Unpaired FID by 0.313 and KID by 0.149, showing that gradient competition between garment reconstruction and person generation intensifies significantly at high resolutions.
In the user study (25 participants × 30 samples, double-blind A/B test, \(p < 0.001\)), DeCo-VTON was preferred over CatVTON by approximately a 3.5:1 ratio, and over Leffa by approximately a 2:1 ratio. The improvements in quantitative metrics indeed translate into user-perceivable quality differences.

Highlights & Insights¶

Paradigm innovation of "decouple, don't denoise": The most elegant insight of this paper is attributing the success of dual-UNet to "structural decoupling of condition and denoising" rather than "more parameters", thereby replicating this advantage in a single-network model using a lightweight inference/training recipe. This analytical framework applies to any conditional generation task utilizing spatial concatenation.
Extreme simplicity and technical elegance: The three design principles do not alter the network structure, with each targetting a specific conflict. GFG simply modifies the construction of the unconditional branch, DL alters only the loss computation region, and CLA performs a variable replacement in the inference phase. Yet, together they successfully bridge the gap from "unsuccessful full-parameter fine-tuning" to "SOTA". This "diagnosis \(\rightarrow\) targeted cure" narrative is an exemplary model of paper storytelling.
Natural explanation of cross-architecture generalization: Two completely different backbones, CatVTON (UNet) and Voost (DiT), encountered the same issue under spatial concatenation. DeCo-VTON's analysis (where Conflicts 2 and 3 stem from spatial concatenation itself rather than CFG) naturally explains this phenomenon. The authors note that DL and CLA can theoretically be transferred directly to non-CFG frameworks (e.g., Flow Matching).

Limitations & Future Work¶

GFG is tied to the CFG mechanism and cannot be directly migrated to architectures that do not use CFG (such as Voost's Flow Matching). Despite the authors' hypothesis that DL and CLA are independent of the guidance scheme and can be directly migrated, this remains to be experimentally verified.
The paper's open-source status is unconfirmed, leaving the experimental reproducibility to be verified.
At high resolutions (\(1024 \times 768\)), DeCo-VTON's SSIM still trails behind Leffa and Voost (0.888 vs. Voost's 0.898). While the paper attributes this to the additional supervision from Voost's VTOFF auxiliary task, it also suggests a potential ceiling in structural fidelity when relying solely on decoupling recipes at high resolution.
The paper's analytical framework has been fully validated in VTON but requires further study to be extended to more generalized spatial concatenation tasks (such as reference-guided inpainting/outpainting)—though the authors explicitly point toward this expansion, and the three-conflict diagnosis naturally fits such tasks.

vs. CatVTON: Both share the same network architecture (SD1.5 Inpainting, 860M). CatVTON reported full-parameter fine-tuning to be ineffective and thus adopted Attn-only training. DeCo-VTON unleashes the potential of full-parameter fine-tuning through the three decoupling principles, reducing the FID from 5.425 to 4.438 (an 18.2% reduction) and the KID from 0.411 to 0.010 (a 97.6% reduction).
vs. Leffa (Dual-UNet SOTA): Leffa uses a 1.8B dual-UNet, whereas DeCo-VTON achieves comparable or superior results with only an 860M single-UNet. This demonstrates that the extra parameters in dual-UNet are not necessary—as long as conditional decoupling is done correctly, a single-network architecture can reach the same standard. This conclusion is highly significant for the engineering deployment of VTON.
vs. Voost (DiT Spatial Concatenation): Voost uses an 11.9B DiT + Flow Matching for spatial concatenation, and also faced the failure of full-parameter fine-tuning (FID 6.351 vs. Attn-only 5.269). DeCo-VTON provides a more structured explanation (the three conflicts) for this phenomenon, which was previously attributed to "overfitting." In high-resolution comparisons, DeCo-VTON (860M) achieves a lower Paired FID (4.741 vs. 5.269) with significantly fewer parameters.
vs. Broader Conditional Generation: The paper analyzes the condition-denoising coupling issue in spatial concatenation within the broader context of conditional injection in diffusion models (cross-attention, ControlNet, spatial concatenation), noting that the latter two naturally evade this issue through "structural separation" and "dedicated branches," respectively. This comparison is helpful for understanding the essential differences among various condition injection methods.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First systematic analysis of the root causes of full-parameter fine-tuning failure under spatial concatenation, proposing an elegant three-principle solution. The idea of visualizing the noise prediction behavior of the dual-UNet reference network itself represents a novel methodological contribution.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensively compared against 12+ methods on two standard datasets, featuring quantitative, qualitative, progressive ablation (validating each principle independently), efficiency analysis, user studies, and cross-architecture high-resolution comparisons. The ablation experiments are logically designed around the three conflicts, presenting a clear causal chain.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation is developed step-by-step with incredibly logical writing: visualizing the network to discover the decoupling insight \(\rightarrow\) analyzing how spatial concatenation violates this insight \(\rightarrow\) deducing the three conflicts \(\rightarrow\) proposing the three corresponding principles \(\rightarrow\) progressive ablation validation. Every design choice is well-justified, and the writing is concise and focused.
Value: ⭐⭐⭐⭐⭐ Beyond proposing an improved VTON model, the paper provides a transferable analytical framework—the "condition and denoising must be decoupled" principle could apply to all spatial-concatenation-based conditional generation tasks (e.g., reference-guided inpainting/outpainting), potentially impacting fields far beyond VTON. It also unifies and explains phenomena previously observed but left unexplained in CatVTON and Voost.