Skip to content

FreeCloth: Free-Form Generation Enhances Challenging Clothed Human Modeling

Conference: CVPR 2025
arXiv: 2411.19942
Code: https://alvinyh.github.io/FreeCloth
Area: Human Understanding
Keywords: Clothed Human Modeling, Free-Form Generation, Linear Blend Skinning, Point Cloud, Loose Clothing

TL;DR

This paper proposes FreeCloth, a hybrid framework that divides the human surface into three regions: "bare", "deformed", and "generated". It models tight clothing using Linear Blend Skinning (LBS) deformation and loose clothing (skirts, dresses) using a free-form generator free from LBS constraints. It achieves State-of-the-Art (SOTA) performance on the ReSynth dataset, significantly outperforming existing methods, especially in loose clothing scenarios.

Background & Motivation

Background: Data-driven clothed human modeling methods typically predict local deformations in the canonical space and then transform these deformations into the pose space via Linear Blend Skinning (LBS). This paradigm performs well on tight-fitting clothing because the garments closely adhere to the human body and conform to skeletal motion.

Limitations of Prior Work: For loose clothing (skirts, dresses), the LBS paradigm fundamentally fails. When clothing is far from the body (such as the hem region between the legs), the mapping from pose space to canonical space (canonicalization) becomes ill-posed due to the lack of clear skeletal correspondence. This causes the point cloud to "tear" into pant-shaped artifacts. While prior works like POP and SkiRT attempt to mitigate this issue through template refinement, they remain inherently limited by the LBS transformation framework.

Key Challenge: LBS provides valuable structural priors (skeletal motion guiding deformation), but for loose parts far from the body, this prior becomes a constraint. Completely abandoning LBS, however, leads to inaccurate modeling in joint regions. A balance must be struck between "leveraging structural priors" and "maintaining expressiveness flexibility".

Goal: How to model loose clothing regions by bypassing LBS constraints while retaining the advantages of LBS in tight-fitting regions?

Key Insight: It is argued that different modeling strategies should be adopted based on the distance between the clothing regions and the body. Tight-fitting regions are heavily influenced by joint motion and are thus suited for LBS, whereas loose regions are less affected by joint motion and are more suitable for direct generation. This hybrid approach is intuitively natural but has not been systematically implemented before.

Core Idea: Divide the human body into three regions: bare, tight-fitting warp, and loose-fitting generate. LBS is used to handle tight clothing deformations, while a free-form point-cloud generator is employed for loose clothing, achieving a hybrid modeling scheme of "deform where deformation is needed, and generate where generation is needed."

Method

Overall Architecture

The input is a posed SMPL-X body and a clothing type, and the output is a complete clothed human point cloud. The pipeline consists of: (1) dividing the body into three regions based on the Clothing-cut Map; (2) directly copying the bare regions; (3) generating \(X^d\) for the blue tight-fitting regions using an LBS deformation network; (4) generating \(X^g\) for the green loose regions using a free-form generator; and (5) merging the three parts to obtain the final complete point cloud \(X\).

Key Designs

  1. Clothing-cut Map:

    • Function: Automatically determine which regions should use LBS deformation and which should use free-form generation.
    • Mechanism: First, isolate bare regions not covered by clothing (head, hands, feet). Then, utilize the SAM foundation model on rendered normal maps to segment loose clothing regions, which are back-projected into 3D space to mark the regions requiring generation. The remaining areas are designated as tight-fitting deformation regions.
    • Design Motivation: Segmentation accuracy directly determines the coordination efficacy of the two branches. Ablation studies demonstrate that without segmentation guidance, skirts and dresses tear in the leg regions because the LBS and the generator conflict in the same area.

  2. LBS Deformation Network (Tight-fitting Regions):

    • Function: Predict pose-dependent deformations for clothing close to the body.
    • Mechanism: PointNet++ is used to extract multi-scale local pose features \(\phi_k^p\) from the posed body, which are then interpolated via barycentric coordinates to obtain continuous local pose encodings \(z_i^p\) for each point. Combined with local and global clothing encodings, a Pose Decoder predicts displacement \(r_i^c\) and normals \(n_i^c\) in the canonical space. Finally, these are converted to the posed space via LBS transformation \(T_i\): \(x_i^d = T_i \cdot (p_i^c + r_i^c)\).
    • Design Motivation: Local pose encodings capture fine-grained wrinkle variations better than global pose encodings, which has been verified by prior works such as CloSET.

  3. Free-Form Generator (Loose Regions):

    • Function: Completely bypass LBS to directly generate loose clothing point clouds in the posed space.
    • Mechanism: Design a structure-aware pose encoding: the human body is divided into \(K_b\) semantic parts, their features are extracted using PointNet++, and max-pooling is applied to fuse them into a global pose encoding \(h^p\). The generator (based on a modified SpareNet) is conditioned on \(h^p\) and the global clothing encoding \(h^g\) to directly generate the point set in the posed space: \(X^g = \mathcal{G}(h^p, h^g)\). The global clothing encoding \(h^g\) is shared with the LBS branch to ensure consistency in the clothing types generated by both branches.
    • Design Motivation: Loose skirt hems are minimally affected by skeletal motion, making this task akin to "conditioned point cloud completion based on the current pose". Part-level pose encodings capture the correlation between skeletal structure and skirt morphology better than a raw global pose vector (ablation studies demonstrate that without this design, the skirt orientation does not match the pose).

Loss & Training

The total loss is a weighted sum of five terms: \(\mathcal{L} = \lambda_{cd}\mathcal{L}_{cd} + \lambda_n\mathcal{L}_n + \lambda_{rd}\mathcal{L}_{rd} + \lambda_{rg}\mathcal{L}_{rg} + \lambda_{col}\mathcal{L}_{col}\).

  • Chamfer Distance \(\mathcal{L}_{cd}\): Bidirectional nearest-point distance between the predicted point cloud and ground truth (GT).
  • Normal Loss \(\mathcal{L}_n\): L1 normal error.
  • Displacement Regularization \(\mathcal{L}_{rd}\): Constrains the deformation of tight-fitting regions to prevent excessive distortion.
  • Clothing Encoding Regularization \(\mathcal{L}_{rg}\): Prevents the encodings from growing too large.
  • Collision Loss \(\mathcal{L}_{col}\): Penalizes generated points that penetrate the body surface using the body's SDF field, defined as \(\max\{\epsilon - d(x_j^g), 0\}\).

End-to-end training is performed, where both branch networks and clothing encodings are jointly optimized.

Key Experimental Results

Main Results

Subject FID↓ (Ours/POP) MSE↓ (Ours/POP) Explanation
All 37.75 / 57.87 2.61 / 2.88 Overall SOTA
felice-004 (Loosest) 42.41 / 66.43 5.24 / 5.80 Large lead in the most challenging scenario
janett-025 27.95 / 52.55 1.92 / 2.02 Significant advantage in long skirts

In the perceptual study, 63.4% of human evaluators preferred the proposed method, and GPT-4o yielded a 56% preference rate. On the two loosest dresses, over 85% of evaluators chose the proposed method.

Ablation Study

Configuration Effect Explanation
LBS deformation only (a) Pant-shaped tearing Verifies the fundamental limitations of LBS
Lower-body generation only (b) Noise and discontinuity in legs Completely discarding LBS is infeasible
W/o collision loss (c) Garments penetrating the body Collision constraints are necessary
W/o part-level encoding (d) Incorrect skirt-hem orientation Structure-aware design is crucial
W/o Clothing-cut Map Tearing and penetration Segmentation guidance is indispensable
Full model (e) Optimal quality The hybrid scheme provides the highest performance ceiling

Key Findings

  • The combination of collision loss and pose enhancement performs the best, indicating that the generator requires physical constraints to prevent implausible geometries.
  • The ablation of the Clothing-cut Map is the most significant: without it, the LBS and the generator experience severe conflicts in overlapping regions.
  • Although FITE is quantitatively competitive, it suffers from "closed surface" artifacts (erroneously closing open skirt hems). The proposed method inherently avoids this issue.

Highlights & Insights

  • Simplicity and Effectiveness of the Hybrid Modeling Paradigm: Instead of pursuing a single method to solve all problems, this approach adopts a divide-and-conquer strategy based on regional characteristics. This idea is simple and intuitive yet highly effective, with the Clothing-cut Map providing a reasonable demarcation as its core.
  • Inspiring Design of Bypassing LBS with Free-Form Generation: Formulating loose clothing modeling as a "conditioned point-cloud completion" task expands the chronological boundaries of clothed human modeling.
  • Transferability of Structure-Aware Pose Encoding: The design of part-level feature extraction combined with Max-Pooling can be applied to other tasks that require understanding the relationship between human parts and attachments, such as hand-held object generation and accessory modeling.

Limitations & Future Work

  • Based on a point cloud representation, the visual quality is constrained by point density, making it difficult to capture extremely fine wrinkle textures.
  • The Clothing-cut Map relies on SAM pre-segmentation, which may require adjustment for novel clothing types.
  • Evaluated only on the synthetic ReSynth dataset, lacking validation on real-world scan datasets.
  • The generator is clothing-specific (leveraging clothing encodings), and its generalization capabilities for virtual try-on remain unverified.
  • Future work could integrate 3D Gaussian Splatting to achieve real-time textured rendering, a direction also mentioned by the authors.
  • vs POP: POP uses LBS deformation across all regions, where loose skirts attempt to "stretch" points between the two legs, causing tearing. FreeCloth fundamentally avoids this issue.
  • vs FITE: FITE learns implicit clothing templates and performs coarse-to-fine LBS on them, but implicit fields struggle with open surfaces, causing the skirt hem to be "closed". FreeCloth's point-cloud generation naturally supports open topologies.
  • vs DPF: DPF completely discards LBS to optimize a smooth deformation field, but it requires impractical frame-by-frame optimization. FreeCloth's hybrid scheme preserves feed-forward inference efficiency while achieving flexibility.

Rating

  • Novelty: ⭐⭐⭐⭐ The hybrid modeling idea is intuitive and natural, although the "regional divide" strategy itself is not completely novel. The core contributions lie in the systematic implementation and the free-form generator design.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Comprehensively ablated with a persuasive perceptual study, though evaluation solely on synthetic datasets is a limitation.
  • Writing Quality: ⭐⭐⭐⭐ Beautifully crafted figures, clear motivation, and well-organized methodology description.
  • Value: ⭐⭐⭐⭐ Provides an effective solution to practical problems in loose clothing modeling and pioneers a new hybrid modeling paradigm.