PhysAnimator: Physics-Guided Generative Cartoon Animation¶

Conference: CVPR 2025
arXiv: 2501.16550
Code: Project Page
Area: 3D Vision
Keywords: Physics simulation, animation generation, deformable body simulation, video diffusion model, sketch-guidance

TL;DR¶

PhysAnimator combines physics simulation (2D deformable body simulation) with data-driven video diffusion models to generate physically plausible and anime-style dynamic animations from static anime illustrations, supporting interactive control via energy strokes and binding points.

Background & Motivation¶

Dynamic effects in hand-drawn animation (such as fluttering hair and wind-blown clothing) are key to enhancing immersion, but traditional hand-drawn methods are extremely laborious and require professional skills.

Limitations of existing automated methods: - Traditional animation tools: Generate deformation animations based on user stroke inputs and geometric constraints, but are typically only applicable to simple line art or layered drawings, making them unsuitable for complex in-the-wild anime illustrations. - Data-driven video generation models (e.g., DynamiCrafter, Motion-I2V): Lack geometric understanding and physical constraints; the predicted optical flow fields often exhibit artifacts, leading to unnatural deformations and degraded visual quality. - Trajectory-driven control methods (e.g., Drag Anything): Easily misinterpret motion trajectories as camera motion. - Physics-based simulation methods (e.g., PhysGen): Restricted to 2D rigid body motion, unable to handle elastic, flowing effects common in anime.

Key Challenge: How to generate high-quality dynamic animations with cartoon-styled exaggerated effects while maintaining physical plausibility?

Method¶

Overall Architecture¶

The pipeline of PhysAnimator consists of three stages: (1) Segmenting target objects of interest using SAM and generating a triangular mesh to perform deformable body simulation in the image space, producing an optical flow sequence; (2) Extracting the sketch of the reference image and warping it using the optical flow, then rendering high-quality frames through a sketch-guided video diffusion model; (3) Optionally applying a data-driven cartoon frame interpolation model to enhance anime-style dynamics.

Key Designs¶

Design 1: Image-Space Deformable Body Simulation — Physically Consistent Motion Generation

Function: Generates physically plausible dynamic motion sequences (optical flow fields) for anime objects.
Mechanism: Uses SAM to segment targets, uniformly samples boundary points along borders, and constructs a triangular mesh via Delaunay triangulation. It adopts the Fixed Corotated constitutive model: \(\Psi(\mathbf{F}) = \mu \|\mathbf{F} - \mathbf{R}\|_F^2 + \frac{\lambda}{2}(\det(\mathbf{F}) - 1)^2\), where \(\mu, \lambda\) are Lamé parameters, and \(\mathbf{R}\) is the rotational component of the deformation gradient \(\mathbf{F}\). Dynamics are solved using Newton's second law \(\frac{d^2\mathbf{x}}{dt^2} = \mathbf{M}^{-1}(\mathbf{f}_{\text{int}} + \mathbf{f}_{\text{ext}})\) and semi-implicit Euler integration.
Design Motivation: Dynamic effects in anime scenes (e.g., fluttering clothes, swaying hair) are essentially elastic deformations. Adopting a deformable body model naturally captures flow and exaggerated motions. Users can control the rigidity/flexibility characteristics of the object by adjusting the Lamé parameters \(\mu, \lambda\).

Design 2: Sketch-Guided Rendering — Texture-Agnostic High-Quality Frame Synthesis

Function: Converts the optical flow dynamics generated by simulation into high-quality, temporally consistent video frames.
Mechanism: Extracts the sketch \(S_0\) of the reference image and generates a dynamic sketch sequence \(S_t = \mathcal{W}(S_0, \mathcal{F}_{0 \rightarrow t}, w_{0 \rightarrow t})\) using forward warping, where the pixel weight is set as \(w(\mathbf{p}) = \|\mathcal{F}_{0 \rightarrow t}(\mathbf{p})\|_2\). The sketch sequence is fed into an SVD model with ControlNet to render colored frames conditioned on the reference image. Gaussian blur is applied to sketches during both training and inference to handle imprecise segmentation.
Design Motivation: Directly warping the original image introduces black-hole artifacts due to occlusion. Sketches, acting as sparse geometric representations, are more robust to imprecise segmentation, while ControlNet leverages generative capabilities to repair imperfections.

Design 3: Complementary Dynamics Enhancement — Data-Driven Anime-Style Supplementation

Function: Discovers and supplements exaggerated anime-style dynamic effects that physical simulation cannot capture.
Mechanism: Selects keyframes every \(n=15\) frames from the sketch-guided rendering outputs and uses the ToonCrafter cartoon frame interpolation model to generate intermediate frames. The control scale is set to 0.1 to balance physical motion and stylized details.
Design Motivation: Dynamic effects in anime do not strictly adhere to physical laws (e.g., exaggerated "squash and stretch"), and 2D simulation cannot fully capture 3D effects. This mimics the workflow of industrial animation pipelines (keyframes first, followed by in-betweening).

Loss & Training¶

The training of sketch-guided ControlNet utilizes the standard LDM denoising loss: \(L_\epsilon = \|\epsilon - \epsilon_\theta(z_t; c, t)\|_2^2\). The training data is obtained from the Sakuga-42M dataset, filtering 380k sketch-video pairs.

Key Experimental Results¶

Main Results: Quantitative Comparison (20 anime images, 200 videos per method)¶

Method	FID ↓	VSVQ ↑	VSTC ↑	VSDD ↑	VSFC ↑
Cinemo	49.5	2.85	2.80	2.42	2.58
DragAnything	148.9	2.77	2.45	2.97	2.52
DynamiCrafter	94.9	2.78	2.68	2.53	2.51
Motion-I2V	121.8	2.70	2.50	2.66	2.39
PhysAnimator	90.4	2.89	2.86	2.48	2.64

User Study: Preference Rate¶

Baseline	Visual Quality	Temporal Consistency	Motion Plausibility	Overall Preference
vs Cinemo	86%	83%	82%	81%
vs DragAnything	93%	91%	89%	91%
vs DynamiCrafter	84%	78%	76%	81%
vs Motion-I2V	95%	94%	97%	96%

Key Findings¶

Cinemo achieves the lowest FID but the videos are almost static, while DragAnything gets a high motion score but is falsely inflated due to misinterpreting trajectories as camera motion.
PhysAnimator leads comprehensively in visual quality, temporal consistency, and factual consistency, with a user preference rate ranging from 76% to 97% in terms of motion plausibility.
Physics simulation ensures geometric consistency, avoiding the common deformation artifacts found in purely data-driven methods.
The sketch-guidance strategy preserves temporal consistency and visual quality better than direct warping-plus-inpainting.

Highlights & Insights¶

Complementary Architecture of Physics Simulation and Generative Models: Physics simulation provides plausible motion while the generative model handles rendering and stylization, leveraging the strengths of both.
Ingenious Choice of Sketch as an Intermediate Representation: Bridges the gap between texture space and geometric space, offering robustness to errors and ease of processing for ControlNet.
Mimicking Industrial Animation Workflows: The keyframe-plus-interpolation approach is natural and aligns with domain knowledge.

Limitations & Future Work¶

It depends on the segmentation quality of SAM; insufficient accuracy degrades both simulation and rendering.
It only supports 2D simulation, unable to fully capture 3D motion effects.
The interactive mechanism of energy strokes still requires manual design; future work could explore automated force-field generation.
It can be extended to support more physical effects (e.g., fluid simulation for smoke and fire).

PhysGen: Image animation based on rigid body physics simulation, but unsuitable for elastic deformation.
Motion-I2V/Drag Anything: Trajectory-controlled video generation, lacking physical constraints.
ToonCrafter: A cartoon keyframe interpolation model, adopted in this work to enhance stylized dynamics.
Insight: Physics simulation does not need to strive for absolute realism; instead, it provides structured guidance for the generative model, which takes care of aesthetic beautification and detail synthesis.

Rating¶

⭐⭐⭐⭐ — Innovatively introduces deformable body physics simulation into cartoon animation generation, featuring a clear pipeline design where each module has distinct responsibilities. The user study results are highly impressive. It holds substantial practical value and lowers the barrier to animation production.