PIAvatar: Physically Interactive Avatars via Deformation Gradient Decoupling¶

Conference: ECCV 2026
arXiv: 2606.21162
Code: None
Area: Human Understanding
Keywords: Physical Simulation, 3D Human Avatar, Deformation Gradient Decoupling, MPM, Skeletal Pose Estimation

TL;DR¶

PIAvatar proposes an MPM-based physically interactive 3D human avatar framework. By explicitly decoupling user-defined kinematic velocity from the deformation gradient update, it eliminates unintended internal stresses generated during motion driving. It also embeds a skeletal structure to achieve real-time tracking of deformed poses through closed-form optimization, supporting bidirectional human-human and human-object physical interactions alongside non-rigid surface deformations simultaneously within a unified MPM simulation framework for the first time.

Background & Motivation¶

3D human avatar generation technologies have made significant progress in recent years, reaching impressive levels of visual fidelity using implicit representations, 3D Gaussian Splatting, and parametric body models (SMPL/SMPL-X). However, a common critical limitation of these models is that they only model geometry and appearance without incorporating physical properties. Consequently, avatars can only perform kinematic animation—"playing" motions according to preset pose sequences—failing to physically respond to external forces such as collisions, pushes, and gravity, and preventing realistic bidirectional physical interactions with other avatars or environmental objects. An avatar kicking a ball cannot actually "kick" the ball, and a pushed avatar will not stumble backward.

In recent years, researchers have attempted to infuse physical properties into avatars. RL-based methods (such as CLOSD, InterMimic) enable avatars to learn interaction policies with objects. However, due to the reliance of large-scale parallel training on simplified physical environments (MuJoCo, Bullet, PhysX), the avatars are simplified into geometric primitives like cylinders, failing to represent non-rigid deformations of muscles, skin, and soft tissues. Simulation-driven approaches have made promising steps toward physical realism, but each exhibits critical gaps: Half-Physics couples a kinematic human model with a physics engine, allowing external forces to change the avatar's pose, but the response is confined to the pose level without surface deformation; PhysAvatar (C-IPC-based) and MPMAvatar focus on garment simulation, supporting the avatar's force application on objects, but backward interactions (objects exerting forces on the avatar to deform it) are weak or missing.

The key challenge lies in the working mechanism of the MPM framework: users drive the avatar's motion by setting particle velocities, which are interpolated to grid momentum during the P2G (particle-to-grid) transfer phase and used to update the deformation gradient $\boldsymbol{F}_p$ during the G2P (grid-to-particle) phase. This change in $\boldsymbol{F}_p$ inevitably generates Cauchy stress $\boldsymbol{\sigma}_p$ according to the constitutive model. In the next frame, this stress is fed back to the grid as an internal force, partially canceling out the motion velocity, which prevents the avatar from accurately reaching the target pose. To put it simply, in standard MPM, "motion drive" and "physical simulation" share the same deformation gradient channel, and kinematic velocity inevitably "contaminates" the stress calculation—this is the core problem addressed in this paper. Additionally, once an avatar undergoes non-rigid deformation during physical interaction, its pose can no longer be directly tracked, typically relying on expensive parametric model fitting or non-linear optimization to recover.

The core idea of this work is to explicitly "divide out" the user-applied kinematic velocity from the deformation gradient, preventing the motion drive from generating internal stress, while fully preserving the stress generated by external contact forces. This simultaneously achieves accurate pose-following and realistic physical interaction within the same MPM framework.

Method¶

Overall Architecture¶

The core problem PIAvatar addresses is how to enable an avatar to accurately follow a user-specified pose sequence (such as kicking or jumping in AMASS) while naturally engaging in bidirectional physical interactions with the environment and others (collisions, deformations, momentum transfer) within an MPM physical simulation framework based on stress constitutive models. The overall approach is a two-pronged strategy: the forward path eliminates parasitic stresses introduced by motion driving through deformation gradient decoupling, while the feedback path guarantees that the deformed, actual pose can still be tracked in real-time using embedded skeletons and used for calculating motion velocities in the next frame.

The inputs of the entire framework are user-specified pose sequences (from datasets like AMASS) and the avatar's particle representations (Animatable Gaussians or sampled particles from a SMPL-X mesh). The outputs are physically simulated particle states (positions, velocities, deformation gradients) for each frame, which can be directly used for rendering. The processing pipeline of each frame is: bone pose extraction $\to$ kinematic velocity calculation $\to$ P2G transfer of kinematic velocity $\to$ deformation gradient decoupling $\to$ standard MPM physics stepping (incorporating multi-field contact handling) $\to$ G2P update of particle states.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Input: Pose Sequence<br/>+ Particle States"] --> B["Skeletal Pose Extraction<br/>Kabsch for RT of Each Bone"]
    B --> C["Kinematic Velocity Calculation<br/>Incremental LBS + Shape Preservation"]
    C --> D["P2G: Kinematic Velocity<br/>Transferred to Independent Grid"]
    D --> E["Deformation Gradient Decoupling<br/>F = F·(F^k)^-1"]
    E --> F["Standard MPM Physics Stepping<br/>Multi-Field Contact Handling"]
    F --> G["G2P: Update Particles<br/>Positions/Velocities/Deformation Gradients"]
    G -->|Next Frame| B

Key Designs¶

1. Deformation Gradient Decoupling: Eliminating Parasitic Stresses from Motion Driving

This is the most critical design of PIAvatar. In standard MPM, P2G and G2P form a closed loop: users set particle velocities $\boldsymbol{v}_p$ $\to$ P2G interpolates to the grid $\to$ G2P updates deformation gradient $\boldsymbol{F}_p \leftarrow (\mathbf{I} + \Delta t \nabla \boldsymbol{v}_p) \boldsymbol{F}_p$ $\to$ stress $\boldsymbol{\sigma}_p = \frac{1}{J_p} \boldsymbol{F}_p \frac{\partial \Psi(\boldsymbol{F}_p)}{\partial \boldsymbol{F}_p}^\top$ is generated based on the constitutive model $\to$ in the next frame's P2G step, this stress acts as an internal force $\boldsymbol{f}_i^{\text{int}} = -\sum_p V_p \boldsymbol{\sigma}_p \nabla w_{ip}$ that cancels out the motion velocity. For non-rigid human body motion, the user-defined kinematic velocity is partially absorbed as accumulation of $\boldsymbol{F}_p$, generating unintended internal stress and preventing the avatar from precisely reaching the target pose. Figure 3 visually illustrates this vicious cycle: kinematic velocity $\to$ deformation gradient changes (blue and red ellipse deformation) $\to$ stress generation $\to$ motion resistance.

PIAvatar's solution is to decompose the particle velocity into a kinematic velocity $\boldsymbol{v}_p^k$ and a physical velocity. Specifically, $\boldsymbol{v}_p^k$ is separately transferred via P2G to an independent kinematic grid to compute its corresponding kinematic velocity gradient $\nabla \boldsymbol{v}_p^k$ and kinematic deformation gradient $\boldsymbol{F}_p^k$:

\[m_i^k = \sum_p w_{ip} m_p^k, \quad m_i^k \boldsymbol{v}_i^k = \sum_p w_{ip} m_p^k \boldsymbol{v}_p^k$$ $$\nabla \boldsymbol{v}_p^k = \sum_i \boldsymbol{v}_i^k (\nabla w_{ip})^\top, \quad \boldsymbol{F}_p^k \leftarrow (\mathbf{I} + \Delta t \nabla \boldsymbol{v}_p^k) \boldsymbol{F}_p^k\]

Then, the kinematic component is removed from the total deformation gradient—this is a first-order approximation operation:

\[\boldsymbol{F}_p \leftarrow \boldsymbol{F}_p (\boldsymbol{F}_p^k)^{-1}\]

After decoupling, only deformations caused by external contact forces are preserved in $\boldsymbol{F}_p$. Consequently, the stress calculated from the constitutive model purely reflects realistic physical interactions and no longer hinders the motion drive. This design shares semantic roots with the elastic-plastic decomposition (Stomakhin et al. 2013) in snow simulation and thin shell stiffness correction (Guo et al. 2018)—both being multiplicative decompositions of $\boldsymbol{F}_p$—but applied in entirely different contexts: previous works decomposed the elastic/plastic components of material constitutive behavior, whereas this work decomposes velocity sources into motion drive and physical response.

Supplementary material Fig. 11 visually validates the necessity and effect of this design with three sets of comparisons: (a) Standard MPM: stress stops the avatar from reaching the target pose, showing almost zero skeletal posture transition; (b) Stress totally removed: the avatar partially follows the motion but completely loses contact response, passing through and fusing with objects; (c) PIAvatar removing only kinematic stress: the avatar accurately follows the motion while pushing objects away and displaying natural collision deformations.

2. Skeleton-Based Pose Regression: Real-Time Closed-Form Tracking of Deformed Poses

Physical interactions inevitably lead to the avatar's non-rigid deformation—such as body denting after being hit by a ball, or pose changes after being pushed by another avatar. This presents a troublesome issue: the actual pose of the deformed avatar is no longer the preset value from the input pose sequence. How do we know what its current "pose" is? In standard MPM, this is an ill-posed inverse problem, typically relying on expensive parametric SMPL fitting (such as SMPLify) or learning-based regression (such as HMR, VIBE).

PIAvatar's solution is very straightforward and efficient: embedding a set of skeletal particles inside the avatar surface (using the OSSO skeleton model, with the original 74,496 particles downsampled by a factor of 10 through voxelization to approximately 7,450 particles to reduce Kabsch computational cost). Each joint corresponds to a set of bone particles—when external forces act on the avatar's surface, the forces are transferred to the bone particles through particle-grid coupling, causing their positions to change. At this point, the Kabsch algorithm is run on each group of bone particles to solve for the optimal rigid transformation $[\mathbf{R}_{\text{bone}}^t, \mathbf{t}_{\text{bone}}^t]$ mapping from the canonical bone particle positions $\boldsymbol{x}_{\text{bone}}^{\text{cano}}$ to their current simulated positions $\boldsymbol{x}_{\text{bone}}^t$:

\[\mathbf{T}_{\text{bone}}^t = [\mathbf{R}_{\text{bone}}^t, \mathbf{t}_{\text{bone}}^t] = K(\boldsymbol{x}_{\text{bone}}^{\text{cano}}, \boldsymbol{x}_{\text{bone}}^t)\]

The Kabsch algorithm is a closed-form least-squares solution: center the two sets of points $\to$ compute the cross-covariance matrix $\mathbf{C} = \mathbf{Q}_{\text{center}}^\top \mathbf{P}_{\text{center}}$ $\to$ SVD decomposition $\to$ $\mathbf{R} = \mathbf{U} \mathbf{V}^\top$ (with reflection correction: if $\det(\mathbf{R}) < 0$, flip the sign of the last column of $\mathbf{U}$) $\to$ $\mathbf{t} = \mathbf{Q}_{\text{mean}} - \mathbf{R} \mathbf{P}_{\text{mean}}$. The entire process requires no iteration and is a deterministic closed-form solution.

Having obtained the current actual pose, the kinematic velocity is computed using incremental LBS: take the incremental transformation $\hat{\mathbf{T}}^{t \to t+1} = (\hat{\mathbf{T}}^t)^{-1} \hat{\mathbf{T}}^{t+1}$ between adjacent frames from the input pose sequence, apply it to the current actual pose to obtain the target pose $\mathbf{T}_{\text{bone}}^{t+1} = \hat{\mathbf{T}}^{t \to t+1} \mathbf{T}_{\text{bone}}^t$, and then compute the two sets of particle positions $\boldsymbol{x}^t = \text{LBS}(\mathbf{T}_{\text{bone}}^t, \boldsymbol{x}^{\text{cano}})$ and $\boldsymbol{x}^{t+1} = \text{LBS}(\mathbf{T}_{\text{bone}}^{t+1}, \boldsymbol{x}^{\text{cano}})$ via LBS. The displacement difference divided by the time step yields the kinematic velocity $\boldsymbol{v}_p^k = (\boldsymbol{x}^{t+1} - \boldsymbol{x}^t) / \Delta t$.

The elegance of the incremental design lies in: even if external forces push the avatar away from the absolute pose defined in the input pose sequence, the motion velocity increment in the next frame remains smooth and reasonable—it only encodes the displacement from the "current position to the next relative pose", rather than jumping from the absolute position in the input sequence. Meanwhile, the skeletal hierarchy guarantees smooth motion propagation and pose consistency among adjacent joints.

3. Shape Preservation Component: Preventing Permanent Deformation after Collisions

In MPM simulation, high-impact collisions may cause surface particles of the avatar to deviate far from the reference LBS surface. Once collision forces cease, these particles may fail to recover voluntarily, leaving permanent depressions or deformations (Supplementary material Fig. 12a). To solve this issue, PIAvatar appends a shape preservation component to the kinematic velocity calculation, inspired by meshless shape matching:

\[v_p^k = \underbrace{\frac{x_p^{\text{LBS},t+1} - x_p^{\text{LBS},t}}{\Delta t}}_{\text{kinematic LBS drive}} + \underbrace{\alpha \frac{x_p^{\text{LBS},t+1} - x_p^{\text{sim},t}}{\Delta t}}_{\text{shape preservation}}, \quad \alpha \in [0, 1]\]

The first term is the standard LBS-driven velocity, which pushes the particles from the current LBS position to the next frame's LBS position. The second term is the shape preservation velocity: when particles deviate from the LBS reference surface due to collision ($x_p^{\text{sim},t} \neq x_p^{\text{LBS},t+1}$), it generates a velocity component pulling the particles back to the target shape, with its intensity controlled by $\alpha$. This is a pure "soft constraint"—instead of hard-resetting particle positions (which would violate physical consistency), it overlays a corrective velocity component so that shape recovery and physical simulation naturally co-exist within the same framework. The authors' experiments demonstrate that with this term, the avatar can restore its original body silhouette after being hit by a ball (Fig. 12c), whereas without it, permanent depressions remain (Fig. 12a).

4. Multi-Field Contact Handling: Resolving Adhesion and Self-Penetration in Multi-Body Interaction

A known limitation of standard MPM is that when particles from multiple objects (or different body parts of the same object) contribute mass and momentum to the same grid node, they are merged into a single average velocity field. This causes two severe problems: (a) after objects touch, the grid no longer distinguishes which momentum belongs to which object, and the computed average velocity traps them together, causing adhesion—e.g., a ball remaining stuck to a foot after being kicked instead of flying away (Fig. 12b); (b) once two body parts of the same avatar (e.g., left and right legs) make contact, their particle momentums are blended, causing them to fuse into a single mass (Fig. 13a) and lose their independent shapes.

PIAvatar adopts the multi-field contact scheme from Bardenhagen et al. to address this issue. The core idea is to maintain independent mass and momentum fields $\{m_k, \mathbf{p}_k\}$ for different objects on each grid node, separately computing independent velocities $\mathbf{v}_k = \mathbf{p}_k / m_k$. For nodes shared by multiple objects, the center-of-mass velocity is first computed as $\mathbf{v}_{\text{cm}} = \frac{\sum_k m_k \mathbf{v}_k}{\sum_k m_k}$. Then, the outward normal of each object is estimated as $\mathbf{n}_k \propto -\nabla m_k$ (the mass gradient direction), and a tension-free contact projection is applied:

\[\tilde{\mathbf{v}}_k = \mathbf{v}_k - \max((\mathbf{v}_k - \mathbf{v}_{\text{cm}}) \cdot \mathbf{n}_k, 0) \mathbf{n}_k\]

This projection is key: it only removes the normal velocity component driving the objects to interpenetrate ($(\mathbf{v}_k - \mathbf{v}_{\text{cm}}) \cdot \mathbf{n}_k > 0$ indicates that the objects are moving toward each other), while fully preserving tangential slip and separating motion. Consequently, approaching objects do not interpenetrate, and separating objects are not "held back"—the contact mechanics are physically correct. For nodes with only a single object, the velocity is not modified at all, keeping the dynamics in non-contact areas identical to standard MPM.

Fig. 12(c) and Fig. 13(b) validate the elimination of adhesion and self-penetration, respectively, after incorporating multi-field contact: once multi-field contact is introduced, the ball detaches normally from the abdomen, and the legs maintain their independent shapes upon contact.

An Illustrative Example¶

Taking the classic interaction scenario of "an avatar kicking a soccer ball" as an example, this details the full simulation pipeline of PIAvatar for a single frame. Suppose the current frame is $t$, and the input pose sequence requires the avatar to perform a right-foot front kick from a standing pose, while a soccer ball weighing 0.5 kg is placed on the ground in front.

Skeletal Pose Extraction: The OSSO bone particles on the avatar's surface are already at some current positions following the physical simulation of the previous frame. For each bone (root bone, spine, left and right upper/lower legs, etc.), take its canonical point set and current simulated point set, and run the Kabsch algorithm—centering the two point sets, SVD-decomposing the cross-covariance matrix, getting $\mathbf{R} = \mathbf{U} \mathbf{V}^\top$, and solving for $\mathbf{t}$—to obtain the rotation $\mathbf{R}_{\text{bone}}^t$ and translation $\mathbf{t}_{\text{bone}}^t$ for each bone in the current frame. This step is a closed-form solution requiring only one SVD per bone. The total Kabsch overhead for the entire skeleton (~55 joints) is about 0.008 ms (single SMPL-X avatar).

Kinematic Velocity Calculation: Extract the transformation matrices $\hat{\mathbf{T}}^t$ and $\hat{\mathbf{T}}^{t+1}$ for frames $t$ and $t+1$ from the pose sequence, and compute the increment $\hat{\mathbf{T}}^{t \to t+1}$. Apply this increment to the current actual pose $\mathbf{T}_{\text{bone}}^t$ to obtain the next frame's target pose $\mathbf{T}_{\text{bone}}^{t+1}$. Compute $\boldsymbol{x}^t$ and $\boldsymbol{x}^{t+1}$ separately via LBS (using standard rigging weights of SMPL-X), and compute each particle's kinematic velocity $\boldsymbol{v}_p^k$ by dividing the displacement difference by $\Delta t$. At this moment, particles in the right foot area acquire a high-speed kinematic velocity toward the soccer ball (e.g., ~3 m/s), while the shape preservation component is superimposed with an intensity of $\alpha=0.3$ to ensure particles can return to the LBS reference surface after collision.

Deformation Gradient Decoupling: $\boldsymbol{v}_p^k$ is separately transferred via P2G to an independent kinematic grid (resolution $200^3$) to calculate $\nabla \boldsymbol{v}_p^k$ and $\boldsymbol{F}_p^k$. Then, the core decoupling operation $\boldsymbol{F}_p \leftarrow \boldsymbol{F}_p (\boldsymbol{F}_p^k)^{-1}$ is executed. At this point, the kinematic component in $\boldsymbol{F}_p$ generated by the right-foot front kick is resolved to the identity matrix (being completely factored out), leaving only contact deformations accumulated from previous frames (e.g., slight compression with the ground while standing).

Physical Interaction and Multi-field Contact: In the P2G stage of standard MPM, the mass/momentum of both the avatar particles and soccer ball particles are transferred to the shared grid. Owing to the multi-field contact processing, grid nodes maintain independent mass/momentum for both the avatar and the ball, computing their respective independent velocities. At contact nodes between the foot and the soccer ball, the velocity difference of the two bodies is processed through contact projection: when the foot moves toward the ball, the normal component is removed to prevent penetration, and the ball receives a forward momentum transfer. The foot area senses an equal and opposite reaction force. Single-object nodes maintain standard MPM behavior.

G2P Update: Grid velocities are transferred back to each object's particles to update positions ($\boldsymbol{x}_p \leftarrow \boldsymbol{x}_p + \Delta t \boldsymbol{v}_p$) and deformation gradients ($\boldsymbol{F}_p \leftarrow (\mathbf{I} + \Delta t \nabla \boldsymbol{v}_p) \boldsymbol{F}_p$). At this moment, $\boldsymbol{F}_p$ only contains deformations from two sources: external contact (slight physical denting on the foot from the foot-ball collision, ball compression) and gravity/ground support. Stresses derive entirely from these realistic physical interactions. The final result: the avatar's right foot precisely reaches the kicking position specified by the input sequence, the soccer ball acquires momentum and flies forward, and the foot and ball separate cleanly without adhesion.

Changing the soccer ball's mass parameter (0.5 kg $\to$ 8 kg bowling ball) yields a completely different physical outcome for the same kicking sequence: the bowling ball barely moves, while the avatar is pushed back by the reaction force, deviating from the target pose—proving that interaction behaviors are fully governed by physical parameters rather than preset motions.

Loss & Training¶

PIAvatar is a pure simulation method and does not involve neural network training; thus, there is no loss function in the traditional sense. The simulation runs at a granularity of 100 MPM substeps per frame on a background grid resolution of $200^3$ (approximately 8M grid nodes), with the time step $\Delta t$ automatically determined by the CFL condition. Kinematic velocity computation and the Kabsch algorithm are executed once per frame (rather than every substep), splitting the total overhead across 100 substeps, while P2G/G2P are executed inside each substep.

The parameter $\alpha$ in the shape preservation term and the material parameters (Young's modulus $E = 10^2 \sim 10^5$ Pa, using Neo-Hookean for the body and Corotated for clothing/hair) are manually configured engineering options calibrated through numerical stability experiments. The Supplementary Material (Sec. 5.4) showcases a preliminary attempt using a VLM (Qwen2.5-VL) + Bayesian Optimization to automatically search for physical parameters (ball density, abdominal stiffness)—using the log-probability differences of the VLM judging the rendered video as "heavy vs. light" or "hard vs. soft" as the reward for BO. The converged results align with human perception (evaluated via an MTurk 50-person user study)—yet this remains an exploratory experiment and is not the core methodology.

Key Experimental Results¶

Main Results¶

Quantitative comparison of pose-following accuracy between PIAvatar and the standard MPM baseline. The experiments cover two avatar representations: Animatable Gaussians (AG, 7 ActorsHQ characters, averaging 300k particles) and SMPL-X (parametric mesh, 10,475 particles). The MSE (mm), RMSE (mm), and [email protected] (proportion of particles with positional error < 0.01m) between the simulated particle positions and the target pose are measured after 100/200/300/400 simulation steps.

Method	Metric	AG 100 steps	AG 200 steps	AG 300 steps	AG 400 steps	SMPL-X 100 steps	SMPL-X 200 steps	SMPL-X 300 steps	SMPL-X 400 steps
MPM Baseline	MSE↓	0.079	0.198	0.275	0.332	0.089	0.138	0.206	0.248
MPM Baseline	[email protected]↑	0.097	0.041	0.020	0.020	0.358	0.181	0.086	0.036
PIAvatar	MSE↓	0.019	0.027	0.036	0.046	0.013	0.022	0.031	0.039
PIAvatar	[email protected]↑	0.844	0.719	0.606	0.534	0.908	0.823	0.670	0.583

Key Findings: PIAvatar comprehensively outperforms the baseline under all steps and representations. For AG at 400 steps, the MSE drops from 0.332 to 0.046 (an 86% reduction), and [email protected] increases from 0.020 to 0.534 (a gain of over 26 times). The critical aspect is that the baseline error continuously accumulates over time (MSE increases 4.2x from 100 to 400 steps), whereas PIAvatar's error increases moderately (only 2.4x), proving that deformation gradient decoupling effectively disrupts the positive feedback loop of stress accumulation and motion deviation.

Ablation Study¶

Since PIAvatar is a physical simulation system rather than a learning-based model (with no detachable neural network modules), ablation is presented through qualitative comparisons in the supplementary materials (Fig. 11-13) and individual component contributions. The table below summarizes the performance of each ablation configuration.

Configuration	Pose Following	Physical Interaction Quality	Key Observations
Standard MPM (No Decoupling)	Poor, unable to reach target	Has contact forces but weak effect	Kinematic velocity generates stress; avatar barely moves (Fig. 11a)
Complete Stress Removal ($\boldsymbol{\sigma}=0$)	Moderate, partial following	None, interpenetration and fusion	Avatar merges and penetrates with objects, losing all physical responses (Fig. 11b)
Decoupled but No Shape Preservation	Good	Good but leaves dents	Surface cannot recover after high-impact collisions (Fig. 12a)
Decoupled but No Multi-Field Contact	Good	Poor, adhesion + self-penetration	Ball sticks to abdomen without detaching (Fig. 12b), legs fuse together (Fig. 13a)
PIAvatar Full	Good, precise following	Good, natural separation	Precise pose + normal collision + shape recovery (Fig. 11c, 12c, 13b)

Skeletal pose tracking accuracy (Table 2) validates the effectiveness of the Kabsch component: the SMPL-X avatar shows a root rotation error of only 0.121 degrees, a relative rotation error of 0.320 degrees, and a distance error of 3.2 mm after 400 simulation steps. The AG avatar exhibits slightly larger errors due to its higher particle count (300k vs. 10k) (relative rotation ~0.6 degrees, distance error ~5.5 mm), yet still remains at a very low level—verifying the stability of closed-form Kabsch skeletal tracking during long-duration simulations.

Runtime analysis (Table 3): A single SMPL-X avatar takes 0.492 ms of total time per substep (compared to 0.170 ms for the baseline MPM), with the overhead primarily stemming from the initialization of the independent kinematic grid and Kabsch computations. The total time for a scene with four AG avatars is 15.259 ms/substep—meaning 100 substeps per frame translates to ~1.5 seconds/frame, leaving room for optimization. The auxiliary memory overhead is approximately 122 MB per avatar (one additional $200^3$ kinematic grid), which is highly acceptable compared to the baseline MPM's 3.8 GB (SMPL-X) or 5.4 GB (AG).

Key Findings¶

Deformation gradient decoupling is absolutely critical: Removing it and reverting to standard MPM causes [email protected] to plummet from 0.844 (AG 100 steps) to 0.097. Parasitic stress is the primary obstacle to driving MPM avatars; decoupling is the only scheme that simultaneously preserves pose accuracy and physical interaction.
Shape preservation and multi-field contact are "engineering necessities" rather than "icing on the cake": Without shape preservation, avatars suffer permanent "disfigurement" after collisions; without multi-field contact, interacting object pairs get "glued together". These two factors dictate the practical performance ceiling of the system.
Interactions are determined by physical parameters, not preset motions: Applying the identical kicking motion to a 0.5 kg soccer ball and an 8 kg bowling ball yields fundamentally different outcomes—the light ball flies away while the avatar remains steady, whereas the heavy ball remains stationary and the avatar is pushed back. This proves that PIAvatar achieves genuine physical interactions rather than mere visual effects.
Heterogeneous material handling is an inherent capability of the framework: By assigning different stiffness values to the body ($E=10^5$), hair ($E=10^2$), and clothing ($E=10^3$), falling objects present distinctly different collision responses depending on which area they strike—driven purely by the MPM constitutive model without any auxiliary modeling.
Soft tissue deformation is an emergent behavior: Belly jiggling naturally emerges during jumping-and-landing processes without the need for explicit soft-tissue modeling—simply by slightly attenuating kinematic velocities in the abdomen area and letting contact stress drive the inter-particle interactions (Fig. 6).

Highlights & Insights¶

The concept of deformation gradient decoupling is exceptionally elegant: The problem originates from $\boldsymbol{F}_p$ conflating all velocity sources, and the solution is a single line of core equation $\boldsymbol{F}_p \leftarrow \boldsymbol{F}_p (\boldsymbol{F}_p^k)^{-1}$—dividing out the kinematic component's contribution. This "separate sources, process independently" philosophy can be migrated to any hybrid particle-grid system that requires simultaneous active control and passive simulation.
Skeletal embedding + Kabsch is an elegant engineering choice: Compared to learning-based pose regression (which requires training data and carries generalization risks) or SMPL fitting (which is iteratively optimized and computationally expensive), the closed-form Kabsch algorithm requires zero training, is deterministic, yields a solution with a single SVD, and incurs only 0.008 ms overhead per frame. Reducing the complex 3D deformed pose tracking problem to bone-level rigid body registration solves the right problem with the right level of abstraction.
The stability wisdom of incremental pose updates: Instead of computing absolute velocities directly from absolute positions in the input pose sequence (which would produce a massive 'snap-back' velocity once an external force pushes the avatar away), the relative increment is computed using $\hat{\mathbf{T}}^{t \to t+1} = (\hat{\mathbf{T}}^t)^{-1} \hat{\mathbf{T}}^{t+1}$ and applied to the current actual pose. This seemingly minor design is crucial—it fundamentally guarantees that kinematic velocity remains smooth under any perturbations.
The inherent elegance of the MPM framework is fully preserved: Bidirectional momentum transfer requires no explicit collision detection or contact force modeling—particles of both the avatar and objects automatically contribute to the same grid nodes, where momentum is conserved and exchanged naturally. PIAvatar's decoupling design does not undermine this core advantage; rather, it enables it to operate effectively in avatar-driven scenarios for the first time.

Limitations & Future Work¶

Scene configuration relies on manual setups: Creating physical interaction scenes requires manually placing the avatar and selecting pose sequences, lacking automated scene understanding or task planning. The AMASS dataset lacks torque control annotations under external forces (e.g., gravity, pushes), and the current simulation does not implement friction, which makes it challenging for the avatar to maintain stable poses like standing after experiencing external forces. The authors suggest incorporating torque control via PID control or RL in the future, as well as introducing friction and richer object properties by interfacing with large-scale physics engines like Genesis.
High-speed collisions still carry risk of adhesion: When two objects collide at extremely high velocities, stress might not build up sufficiently within a few MPM substeps, leading to transient penetration or adhesion. This suggests that learning-based methods could be leveraged to understand scene configurations and object-level features as a complement to pure simulation methods.
Lack of active balance control: Currently, the avatar only executes the input pose sequence, lacking the active control capacity to maintain balance. It simply falls over after being pushed instead of executing self-recovery—a behavior that is unrealistic in human-human interaction scenarios. Integrating physics-based balance controllers (such as an inverted pendulum model with torque compensation) is a foreseeable path forward.
Manual tuning of physical parameters: Material parameters ($E$, density, $\alpha$) rely on manual configuration and trial-and-error. Preliminary results in the supplementary materials using VLM + BO for automated search (Tab. 4-5, validated via user study) demonstrate automation potential, though it remains far from practical in terms of search efficiency and accuracy.
Garment simulation requires coupling external solvers: The secondary dynamics of loose clothing (such as skirts), such as swinging, lagging, and inertial draping, cannot naturally emerge from a volumetric MPM framework, necessitating separate handling via a thin-shell cloth solver (from MPMAvatar). This increases system complexity, and volume-shell coupling can raise issues of numerical stability.

vs. Half-Physics: Half-Physics couples a kinematic human model with a physics engine, allowing external forces to change the avatar's pose, but the response only occurs at the pose level (joint angles) without surface deformation. PIAvatar directly decouples inside the MPM particle framework, simultaneously achieving pose-level response and surface-level non-rigid deformation, offering stronger physical consistency. Half-Physics's pose tracking error is in the centimeter range, whereas PIAvatar's Kabsch scheme is in the millimeter range.
vs. PhysAvatar / MPMAvatar: PhysAvatar (C-IPC framework) and MPMAvatar (MPM framework) both feature garment deformation simulation as their primary selling points. They support avatars exerting forces on clothing/objects, but backward interactions (objects deforming the avatar) are weak or missing. PIAvatar designates bidirectional interaction and non-rigid avatar deformation as the primary goal, leaving garment and soft-tissue deformation as natural byproducts of the framework rather than isolated modules.
vs. RL-based physical avatars (CLOSD / InterMimic / DeepMimic): The core advantage of RL methods is their ability to learn complex task-oriented interaction policies (such as boxing or lifting), at the cost of simplified physical representations (rigid bodies, primitive approximations, joint torque drives). PIAvatar is a pure simulation method with zero training, whose continuum physical consistency is mathematically guaranteed by the MPM framework. A complementary integration of the two—RL learning high-level interaction policies combined with PIAvatar handling low-level continuum simulation—represents an invaluable direction for the physical avatar field.
Relation to broader MPM literature: Multiplicative decomposition of the deformation gradient is not an entirely new concept (e.g., elastoplastic decomposition in snow simulation, thin-shell stiffness correction), but PIAvatar is the first to employ it to decouple two velocity sources: motion drive and physical response—providing a novel application dimension and problem context. The design of skeletal embedding + closed-form Kabsch tracking can be generalized to other particle simulation systems that require real-time pose estimation.

Rating¶

Novelty: ⭐⭐⭐⭐ Applying deformation gradient decoupling to avatar driving is a novel entry point. The combination of skeletal embedding and closed-form Kabsch pose tracking is clean and effective, enabling simultaneous bidirectional interaction and non-rigid deformation within a unified MPM framework for the first time.
Experimental Thoroughness: ⭐⭐⭐⭐ Complete quantitative comparisons (baseline MPM pose-following accuracy), pose tracking accuracy validation, and runtime analysis. Qualitative evaluations cover multiple dimensions, including human-object interaction, human-human interaction, non-rigid deformation, soft tissues, heterogeneous materials, multi-object interaction, and loose clothing.
Writing Quality: ⭐⭐⭐⭐ The problem formulation is exceptionally clear (Fig. 3 and Fig. 11 intuitively illustrate the parasitic stress problem), the methodological logical chain is complete (problem $\to$ decoupling $\to$ skeletal tracking $\to$ engineering refine), and the supplementary materials thoroughly detail implementation specifics (Kabsch pseudo-code, velocity calculation pseudo-code, contact handling).
Value: ⭐⭐⭐⭐ Standardizes a structural gap of 'vision without physics' in the 3D avatar field. It directly promotes application directions such as physical avatar interactions in AR/VR, gaming physical simulations, and digital human simulations. The method is training-free and can run in real-time, lowering the barrier to engineering deployment.