Kirin: Animal Motion Generation from In-the-Wild Video

Conference: ECCV2026
Authors: Brian Nlong Zhao, Zhuoyang Pan, James M. Rehg, Jiajun Wu, Shangzhe Wu
Official Paper: ECCV paper page
PDF: Full paper
Project: https://kirin-ani.github.io/
Area: Human Understanding (articulated animal motion generation)
Keywords: animal motion generation, in-the-wild video, motion diffusion models, image conditioning, automatic rigging

TL;DR

Kirin turns in-the-wild videos into trainable 3D quadruped motion data, uses a reference image to supply morphology missing from text, and applies generated skeletal motion to mesh animation; on AiM3D, image conditioning reduces FID from the text-only model's 11.889 to 6.248, although motion diversity metrics decrease.

Background & Motivation

Animal motion generation is constrained first by data, not just by the generation network. Human motion capture can provide extensive trajectories under a shared skeleton, whereas wild animals are difficult to bring into controlled capture environments and may not behave naturally there. Artist-created or game-derived motions in DeformingThings4D, Truebones Zoo, and AniMo offer clean sequences, but their action coverage and behavioral variation depend on their creators. Internet videos offer broader coverage without the 3D supervision needed to train a skeletal motion generator directly.

Existing video reconstruction methods are not necessarily suited to this objective either. Framewise predictions may approximately match the image while jittering over time; model-free methods may recover appearance without supplying consistent joint definitions across examples. Even with enough 3D motion data, a prompt such as “a dog trotting” leaves body proportions unspecified: a long-legged dog and a short-legged dog should not be treated as having identical morphology. Matching sentences to action categories alone can still produce skeletons that cannot be reliably bound to a particular animal mesh.

Kirin therefore addresses both a unified motion representation from video and motion suitable for the input animal. It uses a SMAL skeleton with sequence optimization for the former and a single image as a morphology condition for the latter, rather than an input video that directly reveals the target motion. Core idea: use a shared skeletal representation reconstructed from real videos as supervision, let text specify behavior and images supply morphology, and connect the resulting motion to renderable meshes through the same skeletal interface.

Method

Overall Architecture

Kirin uses AiM animal videos for training data, while the end user supplies an animal image and an action description. Its output is a textured 3D mesh animation, not newly generated video pixels. The system first performs “Video Motion Extraction” to obtain aligned images, text, and 3D motions. It then trains “Image-Conditioned Motion Diffusion” to denoise motion into skeletal trajectories constrained by text and images. Finally, “Mesh Motion Binding” applies these trajectories to the animal asset generated from the image.

The interface between these parts consists of explicit joints rather than a latent space used only for rendering. Video reconstruction supplies comparable SMAL skeletons, generation learns a distribution of joint motion, and binding handles differences between the target mesh's static pose and the template. Data construction, motion sampling, and mesh deformation are therefore not one jointly trained end-to-end network. Data preprocessing speed should not be mistaken for complete animation generation speed.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["In-the-wild animal videos"] --> B["Video Motion Extraction"]
    B --> C["AiM3D training data"]
    C --> D["Image-Conditioned Motion Diffusion"]
    E["Animal image and text"] --> D
    E --> F["Image-to-3D mesh generation"]
    D --> G["Mesh Motion Binding"]
    F --> G
    G --> H["Textured mesh animation"]

Key Designs

1. Video Motion Extraction: stabilize articulation before restoring global translation

Fitting body shape, joint rotations, and global displacement simultaneously in the original video lets camera changes and animal translation interfere with articulation estimation. Kirin first recovers pose from animal-centered crops. Grounded-SAM2 instance masks and ViTPose++ 2D keypoints provide image constraints, while AniMer initializes SMAL frame by frame. SMAL uses 35 joints and 41 shape coefficients. The framewise shape initializations are averaged into one shape shared across the sequence, and joint rotations are optimized using a continuous six-dimensional representation. Camera parameters remain fixed, preventing the fit from simply changing the camera at every frame to conceal pose errors.

Sequence-level optimization then replaces independent frame predictions. A projection term brings SMAL keypoints closer to detected keypoints, weighted by detection confidence and mask-derived visibility so that occluded or off-image points do not dominate. A smoothness term constrains adjacent rotations and changes in successive rotation increments. The objective is not to make the animal stationary, but to preserve motion while suppressing abrupt changes unsupported by the images. A shared shape also reduces the freedom to change the animal's proportions every frame, although it cannot recover completely invisible leg motion from nothing.

Global translation is estimated separately from the original, uncropped video. SpatialTrackerV2 tracks 3D points on the animal, and their trajectories are averaged to estimate whole-body displacement. Processing includes camera-motion smoothing, ground-plane alignment, residual drift correction using ground points, and temporal smoothing. Initial points are selected inside the animal mask from a 10 × 10 grid in the first frame. Because tracker and SMAL coordinates do not automatically share a scale, the crop size in tracker coordinates is matched to the SMAL head-to-tail distance before translation is combined with the optimized pose. The paper defers the exact scale formula to the supplement, so tracker output should not be interpreted here as validated absolute metric trajectories.

This process produces AiM3D, covering 23 quadruped categories and 29,979 motion sequences: 29,749 for training and 230 for testing. Gemini 2.5 Flash supplies 6 descriptions per video, totaling 179,874 captions. These descriptions increase linguistic coverage without creating additional independent motion samples; the trajectories remain reconstructed supervision rather than motion-capture ground truth. Human inspection of 100 randomly selected test examples judged 86 reconstructions satisfactory and found all captions correct for 90 examples. Dataset scale therefore does not remove the need for cleaning and error analysis.

2. Image-Conditioned Motion Diffusion: constrain motion with both action descriptions and visible morphology

The motion generator adds an image branch to an MDM variant. Frozen DistilBERT encodes the action text, and frozen DINOv3 encodes the reference image. After linear projections align feature dimensions, the global image feature and diffusion timestep feature are broadcast and added to every text token. Each Transformer decoder layer receives this fused sequence through cross-attention, while the noisy motion sequence is progressively denoised. The broadcast relation explicitly described in the paper can be written as:

\[ z_c[i]=z_{\mathrm{text}}[i]+z_{\mathrm{img}}+z_t. \]

Here \(i\) indexes a text token, and all three features have already been projected to the same dimension. The image is not another motion sequence to imitate: it supplies static evidence about body shape, appearance, and viewpoint, while text specifies the action and attributes such as direction. This retains the ability to pool motion patterns across videos while tailoring generation to the animal in the reference image. The fusion itself is lightweight; its main value lies in the conditioning information and training data rather than a new attention operator. It also does not impose jointwise image correspondences, so it should not be taken as evidence of precise morphological measurement.

Text and image conditions are independently dropped with probability 0.2 during training. This enables the conditioning framework to handle text, images, or both and supports classifier-free guidance. Better FID and text–motion alignment do not mean that additional conditioning creates more freedom: fixing morphology narrows the acceptable motion distribution, consistent with the observed reductions in Diversity and Multimodality. An image-only model also lacks a complete action specification and is not an equivalent substitute for text-plus-image generation.

3. Mesh Motion Binding: correct the static bind pose explicitly before transferring motion

The input image also enters the off-the-shelf Rodin image-to-3D system to produce a textured T-pose mesh. That mesh's static pose need not match SMAL's canonical pose, so directly applying rotations could carry the static mismatch into the animation. Kirin first jointly fits SMAL shape and bind pose. Bidirectional Chamfer distance aligns the template surface with the asset, while an edge-length constraint prevents arbitrary stretching merely to match appearance. The fit determines both the asset-specific skeleton shape and each joint's bind transform.

Skinning weights are then transferred from fitted SMAL to the target mesh. Each target vertex finds 10 nearest template vertices and inherits their joint influences through normalized inverse-squared-distance weighting. Generated motion is converted into target joint positions, and joint matching fits framewise SMAL parameters to obtain the driving transforms. Linear blend skinning then deforms the asset:

\[ \tilde v_t=\sum_{j=1}^{J}w_j(v)\,G_{j,t}B_j^{-1}\tilde v. \]

Here \(w_j(v)\) denotes transferred skinning weights, \(G_{j,t}\) the current global joint transform, \(B_j\) the asset's bind transform, and \(\tilde v\) the homogeneous vertex coordinate. The essential step is to remove the asset's static pose through the inverse bind transform before applying the current motion, rather than treating generated joint angles as vertex displacements. The image supplies the appearance asset, the generator supplies skeletal motion, and explicit binding connects them. Mesh-quality problems and template mismatches can still propagate into the animation.

A Worked Example

Consider the paper's example of a moose walking forward. Generation does not require a video corresponding to the reference image. Given the image and sentence, DistilBERT supplies walking semantics, DINOv3 supplies visible morphology, and the diffusion model samples joint motion. In parallel, the same image produces a static 3D mesh. The system fits its SMAL bind pose, transfers skinning weights by interpolation, and drives the mesh using the generated joint transforms at each frame.

This illustrates the image's two roles: it conditions motion and determines the animated object's appearance, without directly specifying the true motion of every future frame. Several walking sequences may be reasonable. They should be assessed through action semantics, distribution quality, and morphological stability rather than exact reproduction of an unseen reference video.

Loss & Training

Video reconstruction uses Adam to optimize shared shape and framewise rotations for 20 epochs at a learning rate of 0.001. The data-term weight is 1.0, the smoothness weight is 100.0, and the rotation-increment smoothing coefficient is 0.2. The paper reports average reconstruction time below 1 second per frame on an NVIDIA A40, including optimization following framewise initialization. This is not the complete system's animation generation latency.

The motion diffusion component uses the standard noise-prediction objective according to the prose, with classifier-free guidance at sampling. Mesh-binding fitting is separate from diffusion training. Several equations in the cached full text contain substantial extraction damage, particularly the diffusion objective and guidance equation. This note therefore describes their purpose from readable prose rather than guessing missing terms, norm exponents, or guidance-scale conventions. Referring to MDM for other settings is also insufficient to infer training steps or the sampling budget.

Key Experimental Results

Main Results

The table below selects means from Tables 1 and 2, omitting confidence intervals. AiM3D uses 230 test motions. AnimalML3D is an external out-of-distribution test set on which none of the methods was trained, containing 1,260 artist-created motions. The paper describes generating 10 motion samples per text and repeating evaluation 10 times, with means and 95% confidence intervals in the original tables.

R-Precision measures top-ranked text–motion retrieval accuracy; FID measures the discrepancy between generated and real motion feature distributions; MM-Dist measures distance in a shared text–motion space. Metric directions are indicated below. Feature distributions and ground-truth references differ between datasets, so absolute FID values across tables should not be used to calculate a multiplicative “generalization drop.” The external-test Kirin row does not explicitly identify its input modality, and its original label is retained here.

Test set	Method and training data	R-Precision Top-1 ↑	FID ↓	MM-Dist ↓
AiM3D	AniMo / AniMo4D	0.029	30.043	8.564
AiM3D	AniMo / AiM3D	0.029	30.516	8.659
AiM3D	Kirin text / AiM3D	0.032	11.889	7.007
AiM3D	Kirin text + image / AiM3D	0.043	6.248	6.218
AnimalML3D	AniMo / AniMo4D	0.031	150.671	17.171
AnimalML3D	AniMo / AiM3D	0.031	155.687	12.334
AnimalML3D	Kirin / AiM3D	0.039	138.145	12.128

On AiM3D, text-plus-image Kirin reduces FID by an absolute 5.641 relative to the text-only model, approximately a 47.4% relative reduction using the latter as the denominator. Top-1 rises from 0.032 to 0.043, a gain of 1.1 percentage points rather than 1.1%. However, the same table gives ground-truth motion a Top-1 score of only 0.113. Absolute values must therefore be interpreted in light of the evaluator's retrieval performance, not equated with human-assessed action correctness.

Ablation Study

Section 4.4 explicitly compares text-only and text-plus-image models with identical training configurations except for the image branch. Diversity is the average distance between independently sampled motions, while Multimodality measures variation across motions generated for the same text. The original table marks both as higher-is-better, but larger values do not automatically imply more realistic motion.

Config	FID ↓	MM-Dist ↓	Diversity ↑	Multimodality ↑
Kirin text, without image branch	11.889	7.007	4.693	4.609
Kirin text + image	6.248	6.218	4.319	4.272
Kirin image only, an input-modality comparison	25.089	Not applicable	1.761	5.804

Reconstruction quality of the training data also matters. The following results come from Table 3. [email protected] is the fraction of projected keypoints within a normalized distance threshold, and MPJVE measures joint-velocity error in projected image space. These are not 3D position errors in motion-capture coordinates.

Reconstruction method	Silhouette IoU ↑	[email protected] ↑	MPJVE ↓	Time ↓
AniMer	0.677	0.537	0.038	<1 s/frame
Kirin	0.698	0.751	0.037	<1 s/frame
SMALify	0.867	0.954	0.023	Approximately 30 s/frame

Key Findings

• Image conditioning improves distribution quality and text–motion alignment, but Diversity falls from 4.693 to 4.319 and Multimodality from 4.609 to 4.272. The improvement involves a trade-off rather than dominance on every metric.
• Dataset benefits cannot be established solely by retraining AniMo: its AiM3D test FID changes from 30.043 to 30.516, and MM-Dist from 8.564 to 8.659. The paper's claim that changing the data already improves performance conflicts with these entries; the numerical results take precedence.
• Reconstruction substantially improves projected keypoints over AniMer while retaining low computational cost, but SMALify remains more accurate on the listed metrics. Below 1 second per frame is also insufficient on its own to establish real-time processing at video frame rates.

Highlights & Insights

• A shared skeletal data interface. Reconstruction, generation, and animation binding use compatible joint representations, bringing the system closer to editable animation assets than image or video quality alone would demonstrate. The benefit comes from the entire data and representation pipeline.
• Static images constrain dynamic generation. Images do not reveal full motion trajectories, but they supply morphology missing from text. The branch ablation demonstrates gains in FID and alignment rather than merely using better-looking appearances for presentation.
• Different errors are corrected separately. Global displacement is tracked in the original video, local pose is optimized in crops, and asset-specific static offsets are corrected by inverse binding. Each step addresses a concrete coordinate or representation mismatch instead of expecting the generator to resolve every inconsistency implicitly.

Limitations & Future Work

• The authors' human inspection identifies missing limb motion when overhead, frontal, or rear views hide the legs, as well as jitter from fast motion or unstable cameras. Only 100 test examples were checked; 86 satisfactory reconstructions do not imply item-by-item validation of the whole dataset.
• The method depends on a SMAL quadruped template. Coverage of 23 categories does not imply support for arbitrary animal skeletons. Generated motion is not validated through contact, dynamics, or biomechanical measurements, so visual plausibility is not equivalent to physical correctness.
• Mesh-animation comparisons with Puppeteer are primarily qualitative. The available main text does not systematically quantify image-to-3D asset quality, binding failure rates, or full-system runtime. Such evidence is still needed to assess practical animation-production utility.
• Better external-test performance than the listed AniMo baselines does not establish that the synthetic-to-real motion gap has disappeared. The main text also contains inconsistencies between tables and narrative claims, while supplementary scale details are absent from the local cache and require verification for reproduction.

Related Work & Insights

• vs AniMo: AniMo uses artist-created game animal motions and a two-stage residual vector quantization model; Kirin uses video-reconstructed supervision and image-conditioned diffusion. The same-data baseline helps expose model differences, but does not attribute every gain exclusively to real-video data.
• vs Ponymation: Both learn motion from online animal videos. Ponymation focuses on horses and unconditional generation, with images mainly supporting appearance reconstruction; Kirin uses images directly for conditional motion generation across categories.
• vs Puppeteer: Puppeteer optimizes mesh animation by matching generated videos. Kirin first generates motion that can drive a skeleton directly, then binds the mesh. The distinction concerns the sources of motion supervision and animation-driving signals, not merely generator size.

Rating

• Novelty: 4/5. The combination of in-the-wild motion data, image conditioning, and asset binding has clear value, although the core network and geometry tools substantially reuse prior methods.
• Experimental Thoroughness: 3/5. In-distribution, external, reconstruction, and modality-ablation evaluations are included, but broader quantitative animation assessment and data-error propagation analysis remain missing.
• Writing Quality: 3/5. The overall pipeline is clear, but some broad improvement claims conflict with the tables, and locally extracted equations should not be used directly for reproduction.
• Value: 4/5. The work offers a practical data route for animal motion modeling without motion capture, within the quadruped template's scope and the evaluated protocols.