VTEdit-Bench: A Comprehensive Benchmark for Multi-Reference Image Editing Models in Virtual Try-On¶

Conference: ECCV2026
arXiv: 2603.11734
Code: https://github.com/Hiuyee124/VTEdit-Bench
Area: Image Generation / Virtual Try-On
Keywords: Virtual Try-On, Multi-Reference Image Editing, Benchmarking, VLM Evaluation, Cross-Task Generalization

TL;DR¶

VTEdit-Bench presents the first benchmark to systematically evaluate the performance of general-purpose multi-reference image editing models in virtual try-on (VTON) scenarios. It includes 24,220 evaluation pairs across 5 progressively complex tasks, and features VTEdit-QA (a reference-aware evaluation agent based on GPT-4o) for fine-grained automatic assessment. Experimental results demonstrate that general image editors match the performance of dedicated models in classic try-on tasks and exhibit stabler generalization in complex scenarios, though a significant gap remains in multi-garment composition control.

Background & Motivation¶

Virtual try-on (VTON) technology has become increasingly important in e-commerce, allowing consumers to visually preview clothing on themselves or virtual models. This plays a significant role in improving purchasing decisions and reducing return rates. In recent years, dedicated VTON models (such as IDM-VTON, CatVTON, and OOTD-Diffusion) have achieved excellent results in the classic Shop2Model (shop image to model image) task through mask-based generation or inpainting techniques. However, real-world scenarios are far more complex than this setup: users may request multi-view displays (Shop2MultiView), multiple individuals wearing the same garment (Shop2MultiModel), clothing extraction from one person's outfit photo to swap onto another (Model2Model), or even combining multiple items like tops, bottoms, bags, and shoes onto the same model (MultiShop2Model). Because dedicated VTON models rely on fixed input assumptions (such as a single person in a frontal pose and a single clothing source) and specific auxiliary conditions (such as OpenPose and human parsing maps), it is difficult to flexibly extend them to these more complex scenarios.

At the same time, general-purpose multi-reference image editing models (e.g., Qwen-Image-Edit-2511, Flux.2, DreamOmni2) have registered rapid progress. Following the image-to-image editing paradigm conditioned on both text and images, they inherently support multi-reference inputs, making their pipelines much cleaner and more flexible than dedicated VTON models. This highlights a potential paradigm shift—general-purpose multi-reference editors might serve as a unified VTON solution, reducing dependence on specialized pipelines. However, there lacks a systematic evaluation of how capable these general-purpose models are on VTON tasks, and where their bottlenecks lie. Existing VTON benchmarks (such as OpenVTON and VTBench) mostly cover only the simplest Shop2Model scenario, which is insufficient to reveal the strengths and limitations of general-purpose editors. Meanwhile, traditional VTON evaluation heavily relies on coarse-grained distribution metrics like FID/KID, which fail to distinguish semantic-level failure modes, such as whether the model identity is preserved or if the clothing texture is correctly migrated.

To address these gaps, the Core Idea of this paper is to design a five-task benchmark called VTEdit-Bench with increasing complexity, accompanied by a VLM-based, reference-aware automatic evaluator called VTEdit-QA (scoring along three dimensions: model consistency, garment consistency, and image quality). This work systematically compares 8 general-purpose multi-reference editing models and 7 dedicated VTON models for the first time, exposing their capability boundaries and evolutionary directions under a unified framework.

Method¶

Overall Architecture¶

VTEdit-Bench is an evaluation benchmark rather than a method. Its core structure consists of three components: a five-task dataset, unified evaluation metrics, and an automated VLM evaluator VTEdit-QA.

The five tasks progress from simple to complex: Shop2Model (classic shop-to-model), Shop2MultiModel (wearing the same garment across multiple people), Shop2MultiView (try-on under non-frontal views), Model2Model (extracting clothing from one person to put on another), and MultiShop2Model (combining multiple items like tops, bottoms, bags, and shoes on the same model). Each task provides unified auxiliary conditions (OpenPose, human parsing, DensePose, garment-free masks, etc.) to guarantee that both dedicated and general-purpose models can be compared under equal input conditions.

The unified evaluation metrics are defined along three dimensions: model consistency (whether the subject's identity and body shape are preserved), garment consistency (whether the clothing texture, color, and boundaries are accurately transferred), and image quality (the overall realism of the generated results and the presence of artifacts). VTEdit-QA wraps these three aspects into the prompts of GPT-4o to output fine-grained scores ranging from 0 to 5, taking the minimum value of the three as the comprehensive score (a conservative strategy, where a failure in any single dimension pulls down the total score).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["VTEdit-Bench<br/>24,220 Test Pairs"] --> B["Five Tasks<br/>Increasing Complexity"]
    B --> C["Shop2Model<br/>9,521 Pairs"]
    B --> D["Shop2MultiModel<br/>2,000 Pairs"]
    B --> E["Shop2MultiView<br/>2,000 Pairs"]
    B --> F["Model2Model<br/>8,299 Pairs"]
    B --> G["MultiShop2Model<br/>2,400 Pairs"]

    H["VTEdit-QA Evaluation"] --> I["Model Consistency<br/>(Identity/Pose/Body)"]
    H --> J["Garment Consistency<br/>(Texture/Color/Boundary)"]
    H --> K["Image Quality<br/>(Realism/Artifacts)"]
    I --> L["Comprehensive Score = min(3 Dimensions)"]
    J --> L
    K --> L

    C --> H
    D --> H
    E --> H
    F --> H
    G --> H

Key Designs¶

1. Progressively Complex Five-Task Design: Covering the Full Spectrum of VTON Scenarios from Classic to Extreme

The benchmark arranges five tasks progressively based on "reference condition complexity". Shop2Model is the starting point: a shop clothing image + a single frontal-view model. Shop2MultiView extends the modality from a single frontal view to multiple views (non-frontal, side, back), testing the robustness of the model to view changes. Shop2MultiModel scales from a single person to multi-person images (group photos of 2 or more people), requiring the model to achieve consistent garment transfer for each individual simultaneously. Model2Model completely removes the constraint of shop images—the clothing source becomes another person's outfit photo, forcing the model to first accurately "extract" the clothing semantics from the source image and then "transfer" them to the target person. The most challenging MultiShop2Model requires combining multiple items such as tops, bottoms, shoes, and bags onto the same person, testing the ability to control occlusion relationships, global coordination, and texture consistency among multiple objects. The data sources and scales of each task vary: Shop2Model is the largest (9,521 pairs from DressCode + VITON-HD + StreetVTON), and the remaining tasks range between 2,000 to 8,299 pairs, ensuring sufficient data while containing annotation costs. This progressive design ensures that each added task exposes a new bottleneck of the models in a specific dimension rather than simply replicating efforts.

2. VTEdit-QA: Reference-Aware VLM Tri-axial Evaluator

To address the limitations of distribution metrics like FID/KID, which fail to identify semantic-level failures (such as identity shifting), VTEdit-QA leverages the visual understanding capability of GPT-4o to score generated results along three dimensions individually. For model consistency, given the source model image and the generated image, GPT-4o judges whether the facial identity, body shape, and pose are preserved. For garment consistency, given the source garment image and the generated image, it judges texture details, color accuracy, and alignment of clothing boundaries with the body. For image quality, given only the generated image itself, it judges whether there are distortions, artifacts, or lighting inconsistencies. Each dimension outputs a score of 0–5, and the comprehensive score takes the minimum of the three (conservative aggregation). The paper validates the reliability of VTEdit-QA over the Spearman Rank Correlation Coefficient (SRCC) between human annotators and GPT-4o scores: inter-human consistency yields 0.75, while the SRCC between GPT-4o and humans reaches 0.72, close to human level. This indicates that VTEdit-QA can replace human evaluation for large-scale automatic evaluation in most scenarios.

3. Unified Auxiliary Condition Protocol: Bridging Input Discrepancies Between Dedicated and General-purpose Models

Dedicated VTON models generally require extra auxiliary conditions (e.g., OpenPose skeletons, human parsing maps, DensePose, and garment-free masks), whereas general-purpose multi-reference editors only need raw images and text instructions. Direct comparison between the two would introduce bias caused by input discrepancies. VTEdit-Bench addresses this by pre-computing and uniformly providing a full set of auxiliary conditions for all test data—offering OpenPose, human parsing, DensePose, and garment-free masks for every model image, and garment masks for every clothing image. In the data filtering stage, the "ability to stably extract these auxiliary signals" is used as a retention criterion, ensuring that the data entering the benchmark is fair for all models under unified conditions. This allows dedicated models to directly run their standard pipelines while general-purpose models run zero-shot (using only unified text instructions), attributing the comparison results to the differences of the models themselves rather than preprocessing variations.

Loss & Training¶

None (benchmark paper, does not involve model training).

Key Experimental Results¶

Main Results¶

Task	Metric	Best Dedicated Model	Best General Model	Explanation
Shop2Model	FID ↓	IDM-VTON (7.40)	Flux.2 (10.88)	General models are close to top dedicated models
Shop2MultiView	FID ↓	IDM-VTON (42.90)	Flux.2 (36.51)	General models outperform dedicated models, which degrade severely
Shop2MultiModel	FID ↓	IDM-VTON (73.90)	Flux.2 (65.81)	General models are more stable
Model2Model	FID ↓	CatVTON (14.98)	Qwen-Edit (13.36)	General models are slightly better, but the gap is minimal
MultiShop2Model	FID ↓	FastFit (11.11)	Flux.2 (15.85)	Dedicated models lead significantly in multi-item combinations
Avg. Rank	—	FastFit (4.8)	Flux.2 (2.2)	General models achieve better overall rank across tasks

VTEdit-QA Comprehensive Score Comparison (Shop-based, higher is better):

Model	Shop2Model	Shop2MultiView	Shop2MultiModel
IDM (Dedicated)	3.60	2.73	1.65
CatVTON (Dedicated)	3.39	2.14	1.26
FastFit (Dedicated)	3.08	1.74	1.18
Flux.2 (General)	3.36	3.38	3.02
Flux.2-klein (General)	3.96	3.99	3.56

Key Findings¶

General-purpose models perform more stably on increasingly difficult Shop tasks. As the complexity increases from Shop2Model \(\rightarrow\) Shop2MultiView \(\rightarrow\) Shop2MultiModel, the FID of dedicated models (e.g., ITA-MDT) aggregates dramatically from 12.23 to 143.61, whereas Flux.2 only rises from 10.88 to 65.81. The comprehensive score of VTEdit-QA also confirms this: IDM's score drops continuously from 3.60 to 1.65, while Flux.2-klein maintains a consistently high level between 3.5 and 4.0.
FID/KID mask semantic-level failures. IDM-VTON achieves the lowest FID (7.40) on Shop2Model, but its VTEdit-QA comprehensive score is only 3.60—lower than Flux.2-klein's 3.96. This is because although IDM's distribution is close to real data on some samples, it exhibits identity shifting or missing garment textures, semantic failures that FID cannot capture.
MultiShop2Model remains a common bottleneck for all models. Multi-garment combination involves complex occlusion reasoning and style coordination. Even the top general-purpose model Flux.2 (comprehensive score 2.25) has a clear gap behind the best dedicated model FastFit (comprehensive score 2.90). DreamO even completely crashes on this task (comprehensive score 0.01).
Model consistency is the biggest weakness of general-purpose models. In the Model2Model task, the model consistency of Qwen-Image-Edit-2511 is only 1.60 (out of 5), substantially lower than CatVTON's 4.44 (dedicated). This shows that general-purpose models easily confuse human identities when "extracting clothing from another person's outfit," carrying facial features of the source person into the generated results.

Highlights & Insights¶

General Editors vs. Dedicated Models Paradigm Comparison: This paper places two entirely different technical routes under a unified benchmark. Results show that general models have a clear advantage in scalability (one model running all 5 tasks), while dedicated models still hold accuracy advantages in specific scenes (such as MultiShop2Model). This provides clear empirical support for the industry to choose technical paradigms.
Generality of the Tri-axial Evaluation in VTEdit-QA: Decomposing generation quality into "Source A Consistency," "Source B Consistency," and "Overall Quality" and aggregating them with the minimum value provides a framework that can be migrated to any generation task involving multi-reference conditions (e.g., face swapping, style transfer, multimodal compositional generation), beyond just try-on scenarios.
Methodology of Progressively Complex Benchmark Design: The five tasks are not randomly selected; instead, each newly introduced task incorporates a new constraint dimension (multi-view, multi-person, multi-item), forming a rigorous "stress test" sequence. This progressive design allows for a finer-grained diagnosis of where a model fails—rather than simply outputting a single aggregated score.

Limitations & Future Work¶

The benchmark currently only covers static image inputs and does not include more complex dynamic scenarios, such as video virtual try-on or clothing tracking under dynamic poses. Video-level assessment can be introduced in the future.
VTEdit-QA currently relies on GPT-4o, which implies API costs and output randomness. Although the paper validates a high correlation with human scores (SRCC 0.72), the fairness of GPT-4o across different body shapes and skin tones warrants further validation.
General editing models are only evaluated under a zero-shot setting (unified text instructions), without testing their performance after prompt optimization or task-specific fine-tuning. In practice, crafted prompt templates for each task might further boost the performance of general models.
The annotation granularity for garment categories in the MultiShop2Model task could be finer—currently it only distinguishes five main categories (tops, bottoms, dresses, shoes, bags), without dedicated analysis on style combinations within the same category (e.g., color and pattern coordination).
The paper lacks user satisfaction studies (e.g., A/B testing) for the generated results; purely automated metrics (even if VTEdit-QA closely aligns with human preferences) cannot fully substitute for the evaluation of real consumer experiences.

vs. Dedicated VTON Models (IDM-VTON / CatVTON / FastFit, etc.): These methods excel in the classic Shop2Model scenario, but are limited by fixed input assumptions and task-specific designs, making them difficult to scale to non-frontal views, multi-person, and multi-item scenes. This paper systematically quantifies this gap.
vs. General-purpose Multi-reference Editing Models (Flux.2 / Qwen-Edit / DreamO, etc.): General models handle all tasks within a unified framework, significantly reducing operational complexity, but still lag behind dedicated models in model consistency (especially identity preservation) and multi-item combination control. These bottlenecks point out directions for next-step methodological improvements.
vs. Existing VTON Benchmarks (OpenVTON / VTBench / StreetVTON): Existing benchmarks cover only a single scenario (e.g., StreetVTON only assesses outdoor scenes) or focus solely on the classic Shop2Model. VTEdit-Bench is the first to systematically cover five representative scenarios, with its progressive complexity being a key innovation.
vs. Other VLM Evaluators (LLM-as-Judge Methods): VTEdit-QA borrows the core concept of utilizing VLMs as evaluators, but innovates by splitting evaluation into three independent dimensions along with a conservative aggregation strategy (taking the minimum instead of the average). This reveals the true weaknesses of models better than single-dimensional scoring.

Rating¶

Novelty: ⭐⭐⭐⭐ The first VTON benchmark to cover general-purpose multi-reference editors. The progressive complexity and VLM tri-axial evaluation are novel designs, though the overall work remains a benchmark.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive comparison of 8 general-purpose models and 7 dedicated models across 5 tasks, combining both FID/KID and VTEdit-QA evaluation systems, along with extra human preference alignment validation.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, intuitive task definition diagrams, and organized experimental analysis (quantitative first, followed by qualitative and insights), offering excellent readability.
Value: ⭐⭐⭐⭐⭐ Fills the gap in systematic VTON evaluations, providing practical reference value for both industry paradigm selection and academia in identifying weak spots of general-purpose models.