Self-supervised Garment Dynamics with Persistent Wrinkles

Conference: ECCV 2026
arXiv: 2606.25065
Code: https://github.com/realcrane/EPNet
Area: Graphics / 3D Vision / Garment Simulation
Keywords: Self-supervised garment simulation, Persistent wrinkles, Elasto-plastic materials, Curriculum learning, Hinge graph GNN

TL;DR

This paper proposes the first self-supervised neural network garment simulator. By leveraging dynamic rest bending energy and physically-inspired curriculum learning, the approach explicitly models the elasto-plastic deformation of fabrics. It generates natural persistent wrinkles for the first time within a self-supervised framework, outperforming existing methods across various garments, body shapes, and motions.

Background & Motivation

Background: Self-supervised neural network garment simulators (such as PBNS, SNUG, NCS, HOOD, SENC) train networks by leveraging physical energy as loss functions, achieving fast inference without training data while maintaining good visual realism. The core mechanism is to formulate garment simulation as a fixed energy minimization problem, where the network predicts the vertex positions of the garment mesh, and the loss function consists of energy terms such as bending, stretching, shearing, collision, and gravity.

Limitations of Prior Work: Prior methods adopt a predefined static Rest Bending (RB) for modeling bending behaviors, assuming that the "zero-energy bending state" of each mesh edge remains fixed. This approach fundamentally treats garments as purely elastic materials, meaning that bending deformations recover completely once external forces are removed, making it impossible to produce "persistent wrinkles" caused by plastic deformation. However, real fabrics exhibit elasto-plasticity: when bending exceeds a yield threshold, irreversible plastic deformation occurs, and creases persist even after the forces are removed. This limitation is the primary reason why the visual quality of existing SSL garment simulators falls far behind that of physics-based simulators (PBS).

Key Challenge: To generate realistic persistent wrinkles, RB must be transformed from a static value to a quantity that dynamically evolves with the deformation history. However, this immediately introduces a "chicken-and-egg" dilemma: the dynamic RB must be predicted by the model, but formulating the loss function itself requires knowing the RB. This mutual dependency transforms the training from a fixed energy minimization task into a "moving target" minimization problem, making direct joint optimization extremely difficult to converge.

Key Insight: The key insight lies in decoupling the elasto-plasticity into an elastic deformation network (E-Net) and a plastic prediction network (P-Net), allowing them to converge alternately through physically-inspired curriculum learning. The core idea of curriculum learning is to transition from easy to hard tasks. First, E-Net is trained on purely elastic deformation (RB near zero). Then, target RBs are computed from the E-Net predictions using physical plasticity rules to train P-Net. Finally, the RB predicted by P-Net is used to re-condition E-Net in the next round. Through this alternation, the system progressively "evolves" from pure elasticity to elasto-plasticity.

Core Idea: To explicitly model fabric plasticity using dynamic RB and a perfect plasticity model, solving the convergence challenge of dynamic loss functions through a dual-network alternating training scheme with curriculum learning, thereby generating natural persistent wrinkles under a self-supervised framework for the first time.

Method

Overall Architecture

This paper constructs a dual-network self-supervised garment simulation system consisting of an elastic network (E-Net) and a plastic network (P-Net). E-Net takes the initial garment state, body motion sequence, and Rest Bending (RB) as inputs to predict the complete time-series garment deformation. P-Net performs message passing on a hinge graph to predict frame-by-frame incremental updates of RB from the current garment state and the RB of the previous frame. The two networks are trained alternately using physically-inspired curriculum learning: the initial RB is set to near zero (purely elastic). After E-Net predicts the deformation, the target RB is computed using physical plasticity rules. P-Net learns to fit this target RB, and then the RB predicted by P-Net serves as the conditional input for the next round of E-Net training, reaching convergence after 4 rounds.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Initial RB<br/>Near Zero"] --> B["E-Net<br/>Predicts Garment Deformation"]
    B --> C["Perfect Plasticity Rules<br/>Compute Target RB"]
    C --> D["P-Net (Hinge Graph GNN)<br/>Predicts RB Sequence"]
    D -->|"Alternating Training x4 Rounds"| B
    D --> E["Output Persistent Wrinkles<br/>Garment Deformation Sequence"]

Key Designs

1. Dynamic Rest Bending and Differentiable Perfect Plasticity Model

Existing methods set Rest Bending (RB) to a globally static value, which fails to capture the accumulation of plasticity caused by multiple bending events. Inspired by perfect plasticity models in physical simulation, this paper defines a differentiable plasticity update rule: when the total strain \(\varepsilon^{(t)}\) exceeds the sum of the plastic strain from the previous frame \(\varepsilon^{(t-1)}_{p}\) and the yield threshold \(\varepsilon_y\), the excess is immediately converted into new plastic strain:

\[ \Delta\varepsilon^{(t)} = \varepsilon^{(t)} - \varepsilon^{(t-1)}_{p} - \varepsilon_y \]

\[ \varepsilon^{(t)}_{p} = \varepsilon^{(t-1)}_{p} + \text{sigmoid}(k_p \Delta\varepsilon^{(t)}) \cdot \Delta\varepsilon^{(t)} \]

where the sigmoid function ensures that the updates are smooth and differentiable. Based on this, the bending energy is formulated as a quadratic form of the elastic strain (total strain minus plastic strain):

\[ W_{bend} = \frac{1}{TD} \sum_t^T \sum_d^D k_b \frac{l^2}{8a} (\varepsilon^{(t)}_{d} - \varepsilon^{(t)}_{p,d})^2 \]

The core advantage of this design is that RB is no longer a predefined, fixed constant, but a spatio-temporal variable that dynamically evolves with the deformation history. Consequently, the fabric can gradually accumulate plastic deformation over multiple bending events along the same edge, producing progressively deeper wrinkles.

2. Physically-Inspired Curriculum Learning

Dynamic RB introduces the "chicken-and-egg" dilemma: RB is required to define the loss function for training E-Net, yet RB can only be calculated from the predictions of a pre-trained E-Net. Directly co-optimizing E-Net and P-Net in practice fails to converge. The curriculum learning scheme proposed in this paper elegantly decomposes the problem into alternating subproblems: ① Initialize RB as near-zero Gaussian noise, making the garment approximately purely elastic; ② Train E-Net using the current RB to predict garment deformation via energy minimization; ③ Starting from the total strain predicted by E-Net, compute the target RB frame-by-frame using the perfect plasticity rules; ④ Train P-Net using the target RB as a supervision signal, enabling it to learn to predict RB from motion history; ⑤ Replace the current RB with the RB predicted by P-Net, and repeat from step ②. As the alternating rounds increase, RB progressively accumulates plastic deformation, guiding the material to transition naturally from pure elasticity to elasto-plasticity. Experiments show that convergence is achieved in just 4 rounds, with E-Net trained for 100 epochs and P-Net for 50 epochs in each round.

3. Hinge-Graph Message Passing P-Net

Persistent wrinkles exhibit clear spatial and temporal correlations: spatially, wrinkles propagate along the mesh neighborhoods; temporally, plasticity accumulates along with the bending deformation. P-Net performs encode-process-decode message passing on a hinge graph derived from the garment mesh. Each node corresponds to an internal edge of the mesh (two adjacent triangles), and each edge connects adjacent hinge nodes. The node features at each time step form an 8-dimensional vector, which includes the current/previous frame's plastic strain, and the hinge midpoint positions in both deformed and rest configurations. Three rounds of message passing (gradually expanding from 1-ring to 2-ring and 3-ring neighborhoods) are employed to incrementally predict the updating of plastic strain for each hinge. Unrolling this frame-by-frame yields the complete RB sequence. Compared to a Transformer, the hinge graph GNN requires only 0.6GB of GPU memory (Transformer needs 9.2GB) and achieves an inference speed of 9.43 FPS (Transformer only 0.17 FPS), making the system highly practical.

A Complete Example: T-shirt Bending Motion

Taking a bending motion sequence from the AMASS dataset (about 200 frames) as an example, the character slowly bends down from an upright posture to 90 degrees and then returns. Round 1: All mesh edge RBs are near zero, and E-Net predicts deformation under the pure elasticity assumption—large bending occurs at the abdominal triangular faces during bending, but the wrinkles disappear completely once returned to the upright posture. Under physical rule updates, edges in the abdominal region exceeding the yield threshold \(\varepsilon_y=0.52\) rad trigger plastic updates, causing the target RB to transition from zero to non-zero. P-Net learns the RB increment patterns of the abdominal hinges. Round 2: E-Net re-predicts conditioned on the non-zero RB—now the abdominal creases remain buckled after bending stops, forming persistent wrinkles consistent with physics-based simulation (PBS). After Round 4 convergence: the wrinkle locations (abdomen, axilla) and sharpness align highly with PBS, reducing the BED by 45.34% compared to the best baseline SENC.

Loss & Training

The loss function is defined as the sum of physical energies, equivalent to solving Newton's second law with an implicit Euler scheme:

\[ \mathcal{L} = \sum_t \frac{h^2}{2} (\frac{\mathbf{x}^{(t+1)}-\mathbf{x}^{(t)}}{h} - \dot{\mathbf{x}}^{(t)})^\top \mathbf{M} (\frac{\mathbf{x}^{(t+1)}-\mathbf{x}^{(t)}}{h} - \dot{\mathbf{x}}^{(t)}) + W^{(t)} \]

where the potential energy is \(W = w_b W_{bend} + w_{st} W_{stretch} + w_{sh} W_{shear} + w_c W_{collision} + W_{gravity}\). The weights are set to \(w_b=4\times10^{-3}\), \(w_{st}=10.0\), \(w_{sh}=1.0\), and \(w_c=10.0\). The yield threshold \(\varepsilon_y=0.52\) rad, the plasticity gain \(k_p=10.0\), and the bending stiffness \(k_b=3.96\times10^{-5}\). E-Net is optimized using the Adam optimizer with lr=1e-4, batch=64, for 100 epochs; P-Net also uses Adam with lr=1e-4, batch=1, for 50 epochs. Materials are sourced from AMASS (252 motion sequences), split into 90% training and 10% testing, sampled at 30 FPS. Complete training takes around 70 hours (on an NVIDIA TITAN RTX), and the inference speed is 0.106 seconds/frame (approximately 9.43 FPS).

Key Experimental Results

Main Results

Table 1: Quantitative comparison on unseen motions (lower is better)

Metric	Ours	Prev. SOTA	Gain
BED(rad)↓	0.28714	0.52529 (SENC)	-45.34%
PE(rad)↓	0.03206	0.12506	-74.36%
BE(rad)↓	0.17118	0.17611 (NCS)	-2.79%
MED(mm)↓	0.02355	0.03191 (SNUG)	-26.19%
CD(m²)↓	0.00027	0.00040 (NCS)	-32.50%
BEN↓	0.02999	0.03412 (NCS)	-12.10%

Table 2: Quantitative comparison on unseen body shapes (trained on Normal, tested on Slim and Obese)

Metric	Ours	Prev. SOTA	Gain
BED(rad)↓	0.46630	0.66125 (SNUG)	-29.48%
PE(rad)↓	0.09690	0.14416	-32.78%
BE(rad)↓	0.19593	0.19755 (NCS)	-0.82%
MED(mm)↓	0.02402	0.02773 (SNUG)	-13.38%
CD(m²)↓	0.00024	0.00032 (NCS)	-25.00%
BEN↓	0.02252	0.02778 (NCS)	-18.93%

Ablation Study

Table 3: Ablation of P-Net architecture (performance of different sequential models on RB prediction)

Architecture	BED↓	PE↓	MED(mm)↓	GPU(GB)↓	FPS↑
RNN	0.86971	0.09968	0.03274	9.1	1.40
LSTM	0.84272	0.07535	0.03251	8.5	1.61
Transformer	0.52095	0.05263	0.02460	9.2	0.17
GNN (Ours)	0.28714	0.03206	0.02355	0.6	9.43

Key Findings

• The hinge-graph GNN of P-Net yields a 44.9% lower BED compared to the Transformer (0.287 vs 0.521), while utilizing only 1/15 of the GPU memory (0.6GB vs 9.2GB) and achieving a 55x increase in FPS (9.43 vs 0.17). This represents not just a minor accuracy improvement, but a qualitative leap from "unusable" to "real-time usable".
• The yield threshold \(\varepsilon_y\) provides intuitive material control: at \(30^\circ\), the fabric behavior is paper-like, easily generating persistent wrinkles; at \(60^\circ\), it is moderate; at \(90^\circ\), it behaves almost as a purely elastic material. Such parameter controls with clear physical meanings cannot be replicated by simply adjusting elastic parameters such as bending stiffness \(k_b\).
• The curriculum learning converges in just 4 rounds: transitioning from pure elasticity (near-zero RB in Round 1) to Round 4 where the predicted RB highly aligns with the target RB, validating the effectiveness of the alternating training scheme.
• In cross-garment generalization tests (training on T-shirt \(\rightarrow\) testing on long sleeves, vests, and pants), the BED only rises from 0.287 (in-distribution) to 0.422. Moreover, the wrinkle sites (unseen regions like elbows and popliteal fossae) remain highly consistent with PBS, indicating excellent generalization capabilities that exceed expectations.

Highlights & Insights

• The design of dynamic RB and the differentiable plasticity rule is highly elegant: by translating the well-established perfect plasticity model from physical simulation into a differentiable neural loss component and using a sigmoid instead of a hard threshold to maintain end-to-end differentiability, it establishes the mathematical foundation that enables the entire method to work.
• Curriculum learning resolves the "moving target" challenge: whereas traditional SSL relies on fixed energy minimization, this work creatively reformulates the problem as "unsupervised tracking of dynamic targets from simple to complex," replacing direct joint optimization with alternating training. This methodology can be transferred to various learning tasks that involve dynamic loss functions.
• The efficiency of the hinge graph GNN is highly educational: conducting message passing on a topological graph formed by the internal edges of the mesh naturally fits the locality of bending without needing global attention, achieving or even exceeding the accuracy of Transformers at an extremely low computational cost.
• Explainability of physical parameters: \(\varepsilon_y\) is a tangible physical parameter (yield strain) rather than an arbitrary hyperparameter. Users can intuitively regulate fabric stiffness through it, which is significantly more user-friendly for artists in the gaming and film industries than tuning loss weights.

Limitations & Future Work

• Inability to model wrinkle flattening due to stretching: The perfect plasticity model only deals with bending plasticity. In reality, stretching fabrics can partially or completely flatten existing wrinkles, a phenomenon that is not modeled in this work.
• Limited performance on loose or draping garments: All SSL methods (including ours) rely on skinning to bind garments to the body. Due to the lack of an explicit contact solver, loose garments cannot naturally detach and drape from the body. This remains one of the most fundamental gaps between current SSL frameworks and PBS, which is also noted as a limitation in the loose draping experiments of the paper.
• Untested multi-layered garments: Inter-layer collision and interaction in multi-layered garments (such as coats over shirts) remain an independent research topic, which is not addressed in this work.
• High training costs: A training time of 70 hours (on a single TITAN RTX) is more than 10 times that of typical data-driven approaches. However, the inference speed is fast (0.106 s/frame), making it suitable for deployment after pre-training.

Related Work & Insights

• vs. PBNS/SNUG/NCS/HOOD/SENC: These methods model garments as purely elastic materials (using static RB), which causes bending energy optimization to completely erase wrinkles once external forces cease. In contrast, this study introduces dynamic RB to explicitly model plasticity, generating persistent wrinkles under an SSL framework for the first time.
• vs. Physics-Based Simulation (PBS): PBS achieves high accuracy by solving contacts and plasticity frame-by-frame, but is computationally expensive. This work integrates a similar physical mechanism into the SSL framework, trading acceptable precision loss for a more than 10x speedup.
• vs. Data-Driven Methods: These approaches require extensive paired simulation data, and their generalization is bounded by the training distribution. This method is entirely self-supervised, demonstrating stronger generalization to unseen body shapes and garments.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Achieves self-supervised persistent wrinkle simulation for garments for the first time, opening up a new direction for elasto-plastic SSL garment simulation.
• Experimental Thoroughness: ⭐⭐⭐⭐ Covers unseen motions, body shapes, and garments, along with architecture ablations and material control analysis. However, the GE/IE metrics in the supplementary materials fluctuate significantly across baselines, and the paper's explanation attributing this to "training randomness" is somewhat weak.
• Writing Quality: ⭐⭐⭐⭐⭐ The motivational chain is clear (pure elasticity \(\rightarrow\) lack of persistent wrinkles \(\rightarrow\) dynamic RB \(\rightarrow\) chicken-and-egg dilemma \(\rightarrow\) curriculum learning \(\rightarrow\) solution), with highly coherent methodological progressions.
• Value: ⭐⭐⭐⭐⭐ Holds direct practical value for garment simulation in digital humans, gaming, and VR/AR, and the curriculum learning framework is highly transferable to other tasks involving dynamic loss functions.

Related Papers

• [CVPR 2026] Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movements
• [ECCV 2024] VideoClusterNet: Self-Supervised and Adaptive Face Clustering for Videos
• [ECCV 2026] SyncCache: Exploiting Asymmetric Dynamics for Fast Audio-Driven Portrait Animation
• [ECCV 2024] Pose-Aware Self-Supervised Learning with Viewpoint Trajectory Regularization
• [ICCV 2025] Bi-Level Optimization for Self-Supervised AI-Generated Face Detection