FitControler: Toward Fit-Aware Virtual Try-On¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/tiny-smart/FitControler
Area: Human Understanding
Keywords: virtual try-on, fit control, garment-shape leakage, multi-scale feature injection, diffusion models

TL;DR¶

FitControler removes cues about the original garment contour, predicts the body-garment layout for a target fit, and injects intermediate features into frozen try-on models, improving fit consistency and image quality on Fit4Men without predicting real garment size or physical comfort.

Background & Motivation¶

Virtual try-on must determine how clothing extends around the body, not merely reproduce patterns, materials, and collars. The same shirt can have a slim, regular, or loose silhouette, while trousers can appear tapered or straight, changing the visual coordination of an outfit. Diffusion-based methods such as CatVTON, Leffa, and FitDiT preserve garment textures well but typically produce a fixed fit for a given input without an independent control interface. Text editing and manual landmark adjustments can modify some attributes, but local styling such as rolling sleeves or tucking a hem is different from the overall ease relationship between clothing and the body. The paper therefore targets image-level fit control rather than fabric simulation driven by precise body measurements.

The challenge is not simply to add a categorical label: existing inputs may already reveal the answer. Conventional try-on training removes the garment region from a dressed-person image and reconstructs the original; a mask drawn along garment boundaries still reveals the original shape. DensePose can also mistake loose clothing edges for body boundaries, allowing a model to reproduce the old silhouette even after garment pixels have been removed. This shortcut helps reconstruct the original fit but resists a requested change, explaining why appending a text prompt can be ineffective. Embedding control directly into a dedicated try-on architecture also limits transfer to models without text input, such as CatVTON, or to different backbones.

FitControler first grounds an abstract fit in a three-class segmentation layout, then uses the features produced while generating that layout to guide garment rendering. This makes the target contour inspectable and avoids relearning the entire body-garment spatial relationship for every try-on model. Fit4Men supplies supervision through explicit menswear fit categories, while contour metrics complement overall image-quality measures such as FID. Core Idea: remove the conditioning shortcut created by the original garment outline, translate fit labels into reusable spatial layout features, and control different frozen try-on models through a lightweight injector.

Method¶

Overall Architecture¶

The inputs are a person image, a product garment image, and a target fit label; the output preserves person and garment appearance while changing the worn silhouette. Shirt labels are slim, regular, and loose, while trouser labels are tapered and straight; the implementation uses categorical labels rather than arbitrary natural-language instructions. The pipeline consists of garment-agnostic preprocessing, fit-aware layout generation, and multi-scale fit injection, followed by image generation using an existing try-on model. The layout is an automatically generated intermediate representation conditioned on the person, garment, and fit, not an additional map that a user must draw.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Person image"] --> PRE["Garment-Agnostic<br/>Preprocessing"]
    PRE --> LAYOUT["Fit-Aware<br/>Layout Generation"]
    COND["Garment image + fit label"] --> LAYOUT
    LAYOUT --> SEG["Three-class segmentation"]
    GT["Training segmentation labels"] -.->|Cross-entropy supervision| SEG
    LAYOUT -->|Multi-scale decoder features| INJECT["Multi-Scale<br/>Fit Injection"]
    INJECT --> VTON["Frozen try-on model"]
    BASE["Base person and garment conditions"] --> VTON
    VTON --> OUT["Target-fit try-on image"]

The segmentation separates background, body, and garment, providing an interpretable layout output; the try-on model receives intermediate layout-decoder features instead. The dashed supervision path is used only during layout training, and inference requires neither target segmentation labels nor a target try-on photograph. The layout generator is trained independently and then frozen for reuse; integration with a particular try-on backbone trains only its matching injector. Thus, the plug-in is not a training-free attachment for arbitrary models: it shares the layout module while requiring a smaller adaptation stage for each target model.

Key Designs¶

1. Garment-Agnostic Preprocessing: removing the contour shortcut in reconstruction training

Instead of following a garment segmentation boundary, the mask is a rectangular region determined by human keypoints. For the upper body, the method uses the horizontal span of shoulder, elbow, and wrist keypoints and the vertical span between shoulders and hips, then expands the region using empirical ratios. Horizontal and vertical padding coefficients are 0.6 and 0.25 for tops; trousers use hip, knee, and ankle keypoints with coefficients of 0.5 and 0.2. The rectangle deliberately ignores the actual sleeve and hem contours, preventing the model from identifying the original fit simply from the mask shape. The coverage margin matters because a mask tightly following the old garment could constrain a requested loose silhouette to the wrong region.

The body condition also needs correction because predicted DensePose can incorporate garment width into body shape. The authors synthesize a dense pose from keypoints and canonical proportions: the torso uses a shoulder-hip quadrilateral with convex upper and lower arcs, while limbs connect circles centered at joints. Arm-joint circle diameters relative to body height are 0.06, 0.048, and 0.033; leg-joint diameters are 0.09, 0.055, and 0.03. Body height is estimated as 3.2 times torso height or 2.3 times leg length, and the synthesized map is intersected with predicted DensePose to constrain the spatial layout. The authors emphasize leakage removal rather than universal optimality of these proportions; they are empirical geometric rules, not a body-measurement model. Canonical proportions can also discard real body-shape differences, and the later diverse-body extension returns to original DensePose, exposing a trade-off between body preservation and leakage suppression.

2. Fit-Aware Layout Generation: making labels alter spatial occupancy first

The layout generator reuses the Stable Diffusion U-Net and its pretrained semantic priors to predict body-garment segmentation. Preprocessing supplies a masked person image, mask, and dense pose; the person, pose, and garment images are encoded into latent space by a VAE. Following CatVTON, the person and garment sides occupy adjacent spatial positions, with zero tensors filling conditions that do not apply to the garment side. Gaussian noise, the downsampled mask, person-garment latents, and pose latents are then concatenated along channels to form the 13-channel input described in the paper. Spatial concatenation and channel concatenation serve different purposes here; this is not simply stacking all images as channels.

The one-hot fit label modulates features through FiLM in each decoder ResNet block. Label-dependent learned scaling and shifting condition normalized intermediate features, allowing identical person and garment inputs to produce different garment occupancy regions. The U-Net encoder remains frozen during training to preserve semantic knowledge, while three-class segmentation supervision trains layout prediction. The authors also collect multi-scale decoder features before the ResNet blocks as the subsequent fit-control representation. These features are not merely segmentation colors: they already encode spatial and semantic information learned while generating the layout, avoiding a separate network that encodes the segmentation again. The cached input and FiLM equations contain missing characters, so this note explains their mechanisms from the prose without presenting reconstructed equations as exact implementations.

3. Multi-Scale Fit Injection: reusing layout features instead of re-encoding a condition image

The injector reconciles differences in channel counts, spatial resolutions, and data structures between layout features and the try-on backbone. At each scale, a zero-initialized convolution aligns features, followed by architecture-specific spatial splitting or interpolation and injection through a ControlNet-style interface. U-Net backbones such as Leffa can match spatial organization through splitting, whereas DiT backbones such as FitDiT require interpolation to a uniform resolution and flattening into a sequence. The shared component is fit semantics, not an assumption that convolutional maps and Transformer tokens have identical tensor shapes. Zero initialization starts the new control path with zero output, avoiding immediate disruption of pretrained generation by untrained features.

Unlike ControlNet or T2I-Adapter, which extract features from a condition image through another encoder, this injector directly uses multi-scale features already computed by the layout generator. Connecting condition generation with condition consumption reduces redundant representation learning and produces faster convergence in the reported ablation. However, lightweight is relative to a full ControlNet condition branch: the injector has 97M trainable parameters in Table 3, exceeding T2I-Adapter's 77M. Both the layout generator and try-on model remain frozen during adaptation, so optimization updates only injector parameters. Original baselines in the main experiment retain their own preprocessing pipelines; these results are not a pure module comparison under identical preprocessing for every method.

A Worked Example¶

Suppose the person originally wears a loose shirt, the product image provides the target garment, and the requested output is slim. A mask following the old garment segmentation already encodes its wide sleeves and torso outline, potentially causing the model to ignore the slim label. The new preprocessor constructs rectangular coverage from shoulder, elbow, and wrist joints and supplies body conditions less affected by clothing width. The layout decoder receives the slim label, changes garment boundaries relative to body regions, and passes the corresponding multi-scale features to the injector. The frozen try-on model renders with product appearance conditioning, aiming to narrow sleeves and torso while preserving patterns, identity, and pose. Switching to regular or loose compares silhouettes for the same person and garment; this illustrates the pipeline rather than reporting a newly measured sample.

Loss & Training¶

The layout generator first learns segmentation using cross-entropy, after which an injector is trained for each try-on model for 7,500 steps. The injector learns to exploit layout conditions through the frozen try-on model's noise-prediction training signal; the norm and exponent in cached Equation 7 are incomplete, so its exact form is not reconstructed here. The paper reports AdamW, batch size 64, learning rate \(1\times10^{-5}\), and FP16 on 4 NVIDIA A6000 GPUs. Original data resolution is \(768\times1024\), while training and evaluation use \(384\times512\); the original resolution should not be mistaken for the experimental output resolution. Fit4Men contains 5,000 shirt pairs and 8,000 trouser pairs, split 4:1 for training and testing; DWPose, DensePose, and Sapiens provide keypoints, dense poses, and segmentation annotations, respectively. The data include different camera distances, front/side/rear views, and poses ranging from standing to partial occlusion, sitting, or running. Inference generates layout features for the requested category and drives the adapted try-on model without person-specific fine-tuning. The main paper does not fully specify layout-training steps, all architectural hyperparameters, or inference sampling settings; the supplied cache also lacks the referenced supplement, so this is not a complete reproduction specification.

Key Experimental Results¶

Main Results¶

Table 2 on page 12 compares models with and without the plug-in on Fit4Men, covering StableVITON, IDM-VTON, CatVTON and its FLUX variant, FitDiT, and Leffa. The selection below shows two backbones and both garment categories in the unpaired setting; lower FID, Hu, and Hd are better, and KID is omitted for compactness. StableVITON is not evaluated on trousers because its available pretrained weights cover only upper-body garments.

Garment	Method	FID	Hu	Hd
Short-sleeve	FitDiT	23.94	0.80	16.89
Short-sleeve	FitDiT + FitControler	18.48	0.46	7.94
Short-sleeve	Leffa	18.77	0.65	10.47
Short-sleeve	Leffa + FitControler	17.73	0.45	7.65
Trousers	FitDiT	15.74	1.86	17.49
Trousers	FitDiT + FitControler	12.67	1.00	8.90
Trousers	Leffa	12.55	1.31	10.63
Trousers	Leffa + FitControler	11.98	0.98	8.80

Hu represents a binary garment contour using 7 Hu invariants derived from normalized central moments, then computes a scalar distance between generated and source contours to describe global shape differences. The readable main text does not specify the exact aggregation or transformation of this vector distance, so no particular norm or logarithmic treatment is assumed here. Hd takes the maximum of the two directed nearest-neighbor contour distances, emphasizing the worst local boundary deviation. The following is the standard definition described by Equation 10, typeset using Euclidean distance rather than recovered from its damaged extraction string:

\[ d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}\|a-b\|_2,\sup_{b\in B}\inf_{a\in A}\|b-a\|_2\right\}. \]

Evaluation generates an image with the same fit label as the source and compares garment contours; this measures shape consistency, not clothing pressure or physical sizing suitability. The check in Table 1 on page 11 supports the metric direction: for a slim source, generating slim yields Hu/Hd of 0.32/6.46, while generating loose yields 0.58/11.61.

Ablation Study¶

Table 3 on page 13 compares condition-injection methods after 7,500 training steps in the paired short-sleeve setting; all values below follow the original table.

Condition Injection	FID	KID	Hu	Hd	Trainable Parameters
ControlNet	13.14	0.91	0.38	6.98	361M
T2I-Adapter	13.50	1.48	0.42	6.97	77M
Fit Injector	12.13	0.49	0.39	6.63	97M

The injector has the best image-quality metrics and Hd, but its Hu is slightly worse than ControlNet, so it does not win every metric. Table 5 on page 14 further compares unpaired short-sleeve results to separate plug-in benefits from additional training; backbone fine-tuning uses the authors' garment-agnostic preprocessor.

Backbone	Config	FID	KID	Hu	Hd
Leffa	Original baseline	18.77	2.59	0.65	10.47
Leffa	Backbone fine-tuning only	18.69	2.18	0.63	12.67
Leffa	FitControler only	17.73	1.54	0.45	7.65
FitDiT	Original baseline	23.94	6.82	0.80	16.89
FitDiT	Backbone fine-tuning only	19.49	3.55	0.81	14.42
FitDiT	FitControler only	18.48	1.86	0.46	7.94

Key Findings¶

FitDiT's unpaired short-sleeve Hd falls from 16.89 to 7.94, a 53.0% reduction reported in the table; the authors attribute its larger benefits to base conditions that leak less fit information.
Additional fine-tuning does not replace fit modeling: Leffa's Hd worsens from 10.47 to 12.67, whereas the plug-in reduces it to 7.65.
In Table 4 on page 13, Leffa trained on 1,000 samples reaches unpaired short-sleeve FID 17.50 and Hd 8.39, versus 17.80 and 8.25 with the full 3,959 samples; small-data performance is comparable, but not every metric improves monotonically.
In Table 6 on page 14, appending fit text and fine-tuning for 16,000 steps still trails explicit layout control, but paired short-sleeve SSIM is 0.868 versus the plug-in's 0.866; stronger overall performance does not imply winning every reconstruction metric.

Highlights & Insights¶

The most useful diagnosis is conditioning leakage rather than insufficient model capacity. A shortcut that helps reconstruction can obstruct counterfactual appearance editing.
A condition generator's intermediate features can themselves serve as a control interface, without compressing them into an image and encoding that image again. This is relevant to systems that predict pose, layout, or depth before image generation.
Realistic appearance and correct contours deserve separate evaluation. Adding Hu and Hd makes sleeve and hem mismatches more visible quantitatively than image-quality metrics alone.

Limitations & Future Work¶

The authors acknowledge that Fit4Men covers only limited fits of men's shirts and trousers, with insufficient body-shape diversity; the results do not establish broad support for womenswear, dresses, or diverse bodies.
Canonical-body preprocessing can sacrifice shape fidelity, and the diverse-body extension using original DensePose still sometimes makes people appear slightly thinner under slim settings.
From a reader's perspective, discrete fit labels and two-dimensional contours provide neither physical ease, fabric elasticity, body measurements, nor comfort estimates; the intended application is visual preview.
Table 4 uses 3,959 as the complete short-sleeve training size, while the dataset section reports an overall 4:1 split; the main text does not explain this particular count difference, so both source statements are retained.
Hu/Hd depend on contour extraction, and the main text does not report statistical agreement with human preferences; body-stratified evaluation, segmentation-error sensitivity, and user assessment could better establish practical control value.

Relationship to CatVTON, Leffa, and FitDiT: these methods provide garment appearance modeling and generation, while FitControler adds layout control rather than replacing their complete rendering architectures.
Difference from ControlNet and T2I-Adapter: generic condition branches encode supplied condition images; this method also generates the fit condition and reuses its features, with Table 3 supporting parameter and quality advantages relative to ControlNet.
Difference from PromptDresser and COTTON: the former uses text semantics and inpainting control, while the latter addresses edits such as garment length; FitControler explicitly models fit as an overall body-garment spatial relationship.
Possible extension, as a reader suggestion: suppress garment leakage while preserving real body shape, then add continuous ease control, which could move closer to personalized try-on than merely expanding discrete fit categories.

Rating¶

Novelty: 4/5. Combines fit control, garment-shape leakage removal, and layout-feature reuse into a cross-backbone plug-in with a clear task focus.
Experimental Thoroughness: 4/5. Includes multiple backbones, paired and unpaired settings, and alternative-mechanism ablations, but body and garment coverage remain limited.
Writing Quality: 4/5. Connects method motivation to the main experiments clearly, although implementation relies partly on the supplement and the full-data count in the sample-size study needs clarification.
Value: 4/5. Adds an interpretable fit-control interface for visual try-on without replacing actual sizing decisions or physical fit validation.