Dress-ED: Instruction-Guided Editing for Virtual Try-On and Try-Off¶

Conference: ECCV 2026
arXiv: 2603.22607
Project Page: https://aimagelab.github.io/Dress-ED/
Code: Available on the project page
Area: Human Understanding / Virtual Try-On
Keywords: Virtual Try-On, Virtual Try-Off, Instruction-Guided Editing, Garment Editing, Multimodal Dataset

TL;DR¶

Dress-ED constructs the first large-scale benchmark dataset (146k verified quadruplets) that unifies VTON (virtual try-on), VTOFF (virtual try-off), and text instruction-guided garment editing. It proposes a unified multimodal diffusion baseline model, Dress-EM, based on MLLM + dual-path connector + DiT, which thoroughly outperforms general-purpose editing models and domain-specific VTON models on instruction-guided garment editing tasks.

Background & Motivation¶

Significant progress has been made recently in virtual try-on (VTON) and virtual try-off (VTOFF), ranging from early warping-field-based garment alignment to attention-conditioned generation using diffusion models. This has substantially improved image realism and the fit of garments on the human body. However, a closer look at existing works reveals a fundamental limitation: all mainstream datasets (such as VITON-HD and Dress Code) are "statically paired," where each sample only records one garment and one person wearing it, and the model's task is merely a fixed "wear-on" or "take-off" mapping. Users cannot use natural language to instruct the model (e.g., "I want to change the color of this garment" or "change the sleeves to three-quarter sleeves"), and there are no edited versions in the dataset to serve as supervision signals. At the same time, although general-purpose image editing datasets (such as InstructPix2Pix, MagicBrush, HQ-Edit, etc.) support text-instruction-guided editing, their samples lack structured garment-to-body correspondences. Without the constraints specific to the fashion domain, it is impossible to guarantee how a garment is worn on the human body and how it changes after editing, both geometrically and semantically. Dedicated fashion editing works (such as EditGarment and T-FIT) only perform garment-to-garment or person-to-person editing, failing to unify the three closely related tasks of VTON, VTOFF, and instruction-guided editing into a single framework.

The Key Challenge arising from this fragmentation is: to achieve controllable instruction-guided fashion editing, a large-scale, semantically consistent "garment - human - editing instruction - edited result" quadruplet training dataset is needed. However, manual annotation of such quadruplets is highly impractical, as every single garment-human pair would require precise editing instructions, an edited garment image, and an edited try-on image, which is extremely costly and difficult to maintain consensus across annotations. The Key Insight of this paper is: can we let models automatically generate these annotated data?

Core Idea: Design an fully automatic pipeline of "MLLM understands garment attributes → Diffusion model performs editing → LLM validates quality" to scale up static garment-human pairs into editing quadruplets with natural language instructions, unifying VTON, VTOFF, and instruction-guided editing in a single framework for training and evaluation for the first time.

Method¶

Overall Architecture¶

The contributions of Dress-ED lie in two aspects: at the data level, it provides an automatic data generation pipeline to construct instruction-guided fashion editing datasets; at the algorithm level, it proposes Dress-EM, a unified MLLM + DiT multimodal diffusion baseline model. The input to the data pipeline consists of the static (I_garment, I_person) pairs already available in the Dress Code dataset, and the output is verified quadruplets with instructions. Dress-EM takes editing instructions and reference images as inputs to directly generate edited try-on or garment images.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Dress Code<br/>Garment-Person Pairs"] --> B["Step 1: Attribute Extraction<br/>Qwen3-VL outputs structured attributes"]
    B --> C["Step 2: Instruction Generation<br/>Rule templates filled for 7 editing types"]
    C --> D["Step 3: Garment Editing<br/>FLUX.2 Klein generates edited garments"]
    D --> E["Step 4: Try-On Rendering<br/>FitDiT wears edited garments back on body"]
    E --> F["Step 5: Quality Verification<br/>GPT-5 scoring -> InternVL distillation"]
    F -->|Score > 80| G["Dress-ED<br/>146k verified quadruplets"]

Key Designs¶

1. Fully Automatic Four-Stage Data Generation Pipeline

The core of the Dress-ED dataset is a four-stage automatic pipeline. In the first stage, Qwen3-VL simultaneously processes the garment image and the try-on person image to extract structured attributes (color, pattern, material, neckline style, sleeve length, etc.) and outputs a description in JSON format. In the second stage, based on the extracted attributes, natural language editing instructions are automatically generated using rule templates, covering seven editing types across two main categories: appearance editing (changing color, pattern, material, fine details) and structural editing (adding details, removing elements, modifying structural cuts). For example, a "blue cotton short-sleeve shirt" might generate instructions like "change the sleeves to long sleeves" or "turn the shirt into yellow." In the third stage, the garment image is first edited with FLUX.2 Klein based on the instruction to generate \(I_{garment\_edit}\), and then the current state-of-the-art VTON model FitDiT is used to put the edited garment back onto the human body to generate \(I_{person\_edit}\). In the fourth stage, GPT-5 scores each (original image, edited image, instruction) triplet (0-100) to evaluate instruction adherence, background preservation, and realism. The GPT-5 scores of ~5k annotated samples are then distilled into InternVL-3.5 to obtain a scale-deployable multimodal validator. This validator is used to filter all ~300k initial samples, retaining only those where both scores are above 80. Consequently, 146,460 high-quality quadruplets are obtained. The ingenuity of this pipeline lies in using a closed loop of "generation + verification" to achieve self-consistent quality control. This ensures that downstream VTON/VTOFF models do not learn the artifacts of generative models—which serves as an excellent application of LLM-as-a-judge and self-training distillation in dataset construction.

2. Connector Architecture with Dual-Path Condition Fusion

Dress-EM is the unified baseline model proposed in this paper, consisting of a frozen InternVL-3.5-8B as a multimodal encoder, a trainable lightweight connector, and a DiT diffusion backbone (based on Stable Diffusion 3 Medium). The input to the VTOFF task is the try-on person image, while the inputs to the VTON task are the masked human image and the garment image. Both tasks, along with the text instruction, are passed into the MLLM for a single forward pass, which outputs a multimodal token sequence fusing the garment, human body, and editing intent. The key innovation lies in the connector: instead of simply pooling the MLLM tokens and feeding them to the DiT, it splits them into two complementary pathways. The Global Guidance Pathway performs masked mean pooling on the MLLM tokens followed by projection through a linear layer, outputting a 2048-dimensional global semantic vector \(g\), which tells the diffusion model "what modifications to make" (i.e., a high-level summary of the editing intent). The Token Refinement Pathway uses lightweight Transformer blocks to perform cross-modal interactive refinement on the complete token sequence, outputting a \(77 \times 4096\) fine-grained multimodal embedding \(z_c\), which tells the diffusion model "where the edited region and context are." The outputs of both pathways, along with the VAE-encoded reference image latent features \(z_{VAE}\), serve as conditional inputs to the DiT. The key insight of this design is that an encoded "high-level intent" is sufficient to guide semantic-level modifications, but insufficient for providing local spatial rendering details; the two pathways are mutually complementary.

3. Reverse Editing Evaluation Protocol

Although this is not a design of the model architecture itself, the paper presents a noteworthy evaluation methodology—the reverse editing protocol. To prevent artificially inflated metrics caused by reference images originating from synthetic models, the ground truth during the test phase is directly obtained from real images: real images are first modified to edited versions using editing instructions (forward editing), and the model is then tasked with executing reverse instructions to restore the edited images back to the original real images (reverse editing). Quantifiable metrics such as FID / DINO-I are calculated along this "synthetic \(\rightarrow\) real" channel. This means all reference evaluations are based on real pixel distributions and are free from distribution shifts of synthetic data. The comparison between forward and reverse editing results is almost consistent, which further demonstrates that the model learns actual semantic editing capabilities rather than reversing synthesis artifacts.

Loss & Training¶

The standard diffusion noise prediction loss is adopted. During training, VAE and InternVL are frozen, and only the connector and DiT are updated end-to-end. A unique aspect is the hybrid supervision strategy: both synthetic edited images and original real images are used as denoising targets. This forces the model to follow instructions for appearance/structural modifications while preserving fidelity to real textures. Using the AdamW optimizer, a cosine learning rate scheduler (initial 1e-4), a batch size of 4, the model is trained for 50k steps on 16 A100 GPUs in DeepSpeed/bf16 precision.

Key Experimental Results¶

Main Results¶

Task	Method	SSIM ↑	LPIPS ↓	FID ↓	DINO-I ↑
Paired VTON	FLUX.2 Klein	0.9291	0.0743	4.32	0.9302
Paired VTON	Qwen-Image-Edit	0.9173	0.0963	8.51	0.8802
Paired VTON	CatVTON	0.9314	0.0801	4.57	0.9144
Paired VTON	Any2AnyTryon	0.9429	0.0702	5.34	0.9074
Paired VTON	Dress-EM	0.9417	0.0628	3.79	0.9383
Unpaired VTON	Dress-EM	—	—	5.82	0.6476
VTOFF	FLUX.2 Klein	0.8694	0.2800	17.45	0.7468
VTOFF	CatVTON	0.8800	0.2415	6.78	0.7918
VTOFF	Dress-EM	0.8800	0.2060	5.06	0.8071

Fine-grained Analysis by Editing Type¶

Dress-EM consistently achieves the lowest DISTS and the highest DINO-I across all seven editing types. The most significant improvements are observed in appearance editing (changing color, pattern, material). For instance, in VTOFF scenarios, the DISTS for color editing with Dress-EM (0.180) is 0.03 lower than that of CatVTON (0.210), demonstrating that multimodal conditional fusion is particularly effective for texture/shade-related editing representations. Although the gains for structural editing (adding details, modifying structural cuts) are slightly smaller, they are equally consistent.

Key Findings¶

A comparison with the commercial model GPT-Image-1.5 on a 100-sample subset reveals that when the strongest general-domain editing model is transferred to fashion editing in a zero-shot setting, its metrics lag behind Dress-EM across the board (LPIPS 0.0838 vs. 0.0622), highlighting the importance of domain-specific datasets and training.
The results of forward editing and reverse editing are almost identical (DINO-I: 0.9493 for forward vs. 0.9383 for reverse), illustrating that the model learns semantic editing instead of merely reversing synthetic artifacts.
Human-machine evaluation indicates that the fine-tuned InternVL validator achieves a 95.6% accuracy alignment with human judgment (77.7% of samples were consistently judged as good by both, and 17.9% were consistently judged as poor), confirming the feasibility of using MLLM-based filtering to replace manual screening.

Highlights & Insights¶

The automated data pipeline with dual-model verification is the biggest highlight: Resolving the data scarcity bottleneck through an automatic pipeline of "MLLM understanding \(\rightarrow\) Diffusion editing \(\rightarrow\) Distilled verification", and distilling the validator from GPT-5 to the open-source InternVL allows scalable reuse. This paradigm is not restricted to fashion editing but can be naturally extended to any domain that requires large-scale paired editing data but suffers from a lack of manual annotations (e.g., medical image editing, product design editing).
Reverse editing evaluation protocol: Utilizing "synthetic \(\rightarrow\) restore to real" for evaluation effectively isolates metric contamination caused by synthetic domain shifts. The approach is simple yet rigorous and is worth promoting in generative evaluations.
The dual-path connector decouples "high-level editing intent" from "fine-grained spatial context." Compared to simple pooling or single-path projection, it provides the DiT with clearer conditional signals of "what to edit + where to edit."
The quadruplet structure of Dress-ED is naturally compatible with various downstream tasks—editing VTON, editing VTOFF, human body editing, garment editing, and standard VTON/VTOFF by simply dropping the instructions, maximizing data utility.

Limitations & Future Work¶

Data is entirely synthesized by models: While intentionally designed, this is also the biggest limitation—the performance upper bound of FLUX.2 Klein and FitDiT determines the data quality ceiling. Although the validator filters out low-scoring samples, gaps may still persist between synthetic and real-world distributions, particularly regarding texture details and lighting consistency. Future work could introduce a small amount of real-world data for domain adaptation or human feedback loops.
Limited garment categories and editing types: The dataset only covers three main categories (dresses, tops, bottoms), omitting accessories (scarves, jewelry) and layered clothing (coats + hoodies). While the seven editing types provide decent coverage, the template-based instruction generation limits language expression diversity—actual users might say "I want it to look more streetwear-styled" rather than just "change color to green."
Dress-EM as a baseline: Restricted by freezing the MLLM and VAE during training, the model capacity is bound by the representation capability of pre-trained features. There was no attempt at joint fine-tuning of the MLLM to better adapt to fine-grained editing tasks.
Extensible directions: Introducing more diverse instruction variants (e.g., using LLMs to freely rephrase template instructions); extending to multi-garment/multi-person interaction scenarios; and upgrading the validator from discriminative scoring to providing fine-grained feedback on "what went wrong."

vs. CatVTON / Any2AnyTryon: These are standard VTON/VTOFF models that do not natively support text instructions. In this study, they were retrained on Dress-ED and adapted for instruction conditioning. Dress-EM consistently leads in all metrics, showing considerable improvements particularly in FID (paired VTON 3.79 vs. CatVTON 4.57) and DINO-I (0.9383 vs. 0.9144), validating the effectiveness of MLLM prior encoding combined with dual-path conditioning.
vs. EditGarment / T-FIT: Prior works in fashion editing only execute single-direction editing from garment-to-garment (EditGarment) or human-to-human (T-FIT), and are unable to process VTON + VTOFF in a unified manner. Dress-ED not only unifies both but also provides the large-scale data foundation required for training and evaluation.
vs. InstructPix2Pix / FLUX.2 Klein: When general-purpose image editing models handle fashion editing in a zero-shot setting, the FID for VTOFF degrades significantly (FID > 17), revealing their lack of domain-specific understanding of garment structures. This, in turn, underscores the necessity of Dress-ED as a domain benchmark.

Rating¶

Novelty: ★★★★☆ [Takes the lead in presenting a unified large-scale benchmark dataset for VTON/VTOFF/instruction-guided editing. The self-constructed automatic generation + verification pipeline holds great potential for broader applications. However, the core components (MLLM, diffusion models, LLM Judge) are highly effective combinations of existing capabilities rather than fundamentally new methods.]
Experimental Thoroughness: ★★★★★ [Comprehensive quantitative evaluations covering paired/unpaired VTON and VTOFF, including fine-grained analysis on 7 editing types, forward/reverse editing comparisons, comparisons with commercial models, and user studies. Metrics span FID, KID, SSIM, LPIPS, DISTS, and DINO-I.]
Writing Quality: ★★★★☆ [The structure is clear, and the motivations and data construction process are described in rich detail. The appendix supplies complete template rules, an attribute bank, and filter ablations, accompanied by rich illustrations.]
Value: ★★★★★ [Dress-ED fills the benchmark void in instruction-guided fashion editing. The 146k high-quality samples and rigorous evaluation protocols will drive VTON/VTOFF forward from "fixed mapping" to "controllable interactive editing," offering clear practical value for e-commerce and fashion AI.]