Skip to content

Forecasting Animal Motion

Conference: ECCV 2026
Paper: ECCV Paper Page
Project: https://motion-forecasting.github.io/
Area: Time Series
Keywords: Animal Motion Forecasting, Point Trajectories, Diffusion Transformer, Non-Rigid Motion, Behavior Modeling

TL;DR

By establishing dense point trajectories as visual tokens for behavior modeling and combining local DINOv3 visual features with a camera motion compensation pipeline, this work introduces a diffusion transformer (DiT) architecture over unordered trajectory sets to forecast complex non-rigid wild animal motion across diverse species without category-specific 3D priors.

Background & Motivation

Predicting the future motion and behavioral dynamics of dynamic agents is a core prerequisite for embodied intelligence and computer vision systems. In unconstrained natural environments, whether during foraging, hunting, vigilance, or social interaction, anticipating the future trajectories and non-rigid deformations of living agents is crucial for situational reasoning and survival. However, while static visual recognition and image generation have witnessed unprecedented breakthroughs, predictive behavioral intelligence remains one of the least understood frontiers in vision. A major root cause of this lag is the absence of an appropriate, general-purpose mid-level representation for behavior—analogous to how discrete tokens structure natural language prediction. Existing paradigms are polarized into two extremes: one directly generates future RGB pixels via large video diffusion models, which conflates fine-grained biological articulation with texture, ambient illumination, shadow drift, and severe camera movements, rendering the optimization problem astronomically data-inefficient and prone to physical hallucinations; the other extreme relies on category-specific parameterized 3D models (such as SMPL for humans or articulated 3D meshes for a handful of domestic species), which depend on rigid skeletal templates, cannot scale to thousands of wild species across the long tail of nature, and fundamentally overlook subtle non-rigid surface deformations.

Overcoming the tension between the entangled data inefficiency of pixel forecasting and the fragile specificity of 3D parametric meshes demands an intermediate representation that injects structural physical constraints without sacrificing cross-species generality. Classical ethology and perceptual psychology (dating back to Johansson's biological motion experiments with moving light displays) demonstrate that while sparse points lack semantic meaning in static poses, their collective relative spatiotemporal trajectories reveal rich 3D topology, articulation mechanics, and behavioral intent. Treating evolving 2D dense point tracks as visual tokens for behavior completely unbinds motion dynamics from appearance and lighting variations, allowing generative capacity to concentrate entirely on motion kinematics. Furthermore, animal movement in natural habitats is driven by multiplicative environmental interactions, following a log-normal distribution rather than a standard power law, and is consistently challenged by intermittent occlusions and unconstrained viewpoints.

Motivated by the structural efficiency of point trajectories for non-rigid dynamics, this paper introduces a generative forecasting paradigm that establishes dense point tracks as visual tokens for animal behavior. The authors construct an automated pipeline featuring camera motion compensation and instance tracking from raw unconstrained nature documentaries, yielding stabilized world-approximate coordinates, and build an occlusion-aware Diffusion Transformer (DiT) that jointly models trajectory dynamics, visibility states, and localized visual context. Core idea: treat dense point trajectories as visual tokens for behavior to disentangle appearance from motion dynamics, using a Diffusion Transformer (DiT) over unordered trajectory sets combined with occlusion-aware velocity reparameterization and camera stabilization to achieve data-efficient, high-fidelity, and cross-species generalizable non-rigid behavior forecasting.

Method

Overall Architecture

The framework is designed to take a single RGB observation frame at \(t=1\), a brief observed motion history of points with their visibility states over \(T_c\) initial frames, and an optional high-level displacement guidance vector \(d\), and forecast the future 2D spatiotemporal trajectories along with binary occlusion indicators for all animal surface points across \(T_c+1\) to \(T\).

The overall pipeline comprises an offline video processing pipeline and an online conditional diffusion generation network. The processing stage tracks dense points across raw videos, filters dynamic animal masks, and solves for background homographies via RANSAC to cancel camera pan and zoom, projecting trajectories into a normalized, camera-stabilized reference frame. The generative network then abstracts each surface trajectory into an individual token that concatenates local DINOv3 visual priors, historical velocity embeddings, initial point spatial positional encodings, and noisy diffusion target velocities. These tokens are processed as an unordered set by a Diffusion Transformer (DiT) through self-attention, and the denoised velocities are integrated along time to reconstruct physical trajectories.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: First Frame RGB Image<br/>+ Observed History Tracks and Occlusion States"] --> B["Video Preprocessing & Camera Stabilization<br/>RANSAC Homography Decoupling Camera from Animal Motion"]
    B --> C["Behavioral Trajectory Token Construction<br/>Concat DINOv3 Feature / Motion History / Noisy Future"]
    C --> D["Occlusion-Aware Velocity Reparameterization<br/>Temporal Velocity Differences + Occlusion Interpolation"]
    D --> E["Trajectory Diffusion Transformer Denoising<br/>Initial Coordinate Position Encoding + DiT Self-Attention"]
    E --> F["Output: Clean Future Trajectories<br/>+ Dynamic Occlusion States (Displacement-Controllable)"]

Key Designs

1. Video Preprocessing & Camera Stabilization: decoupling camera motion and normalizing world coordinates Natural wildlife documentaries are dominated by professional panning, zooming, tracking shots, and handheld tremors. Training a motion prediction model directly in uncorrected pixel coordinates forces the network to memorize extrinsic camera motion rather than intrinsic biological dynamics. To approximate an invariant world coordinate system without access to true 3D depth sensors, the authors design a robust perception pipeline. The pipeline first extracts dense point tracks across raw video sequences using BootsTAPIR and identifies coherent shots by thresholding the proportion of smoothly moving tracks across consecutive frames. Within each isolated shot, Grounding-DINO detects animal bounding boxes and VideoSAM tracks the non-rigid instance masks over time. Crucially, to isolate camera motion, all tracks situated within the animal mask are masked out, and RANSAC is executed on the remaining background point tracks to estimate a frame-to-frame background homography \(H_t\). The animal bounding box in the initial frame is expanded by 50% on each margin to serve as a canonical reference canvas, and subsequent coordinates are transformed via \(H_1 H_t^{-1}\) to undo camera scaling and translation, normalized to \([0, 1]^2\). During point sampling on the animal body, sampling is biased toward thin structures (such as limbs, ears, and tails) to prevent sparse representations on articulated extremities. The resulting 300-hour MammalMotion dataset reveals that animal motion magnitudes fit a log-normal distribution significantly better than a power-law distribution, indicating that wild behavior arises from multiplicative interactions among independent environmental and physiological factors.

2. Behavioral Trajectory Token Construction: fusing local DINOv3 semantics and temporal history The collection of surface points is fundamentally an unordered set lacking a fixed 2D grid structure or pre-defined skeleton topology. To equip each spatially distributed point track with precise anatomical semantics and local geometric context, the architecture designs an expressive per-track token representation. On the first frame crop \(I\), the model extracts visual feature representations using a frozen DINOv3 backbone, which encapsulates rich self-supervised priors about animal parts. For each point track \(n\) starting at initial coordinate \((x_1^n, y_1^n)\), a localized visual feature \(f_n^{\text{DINO}}\) is obtained via bilinear interpolation. The model then encodes the clean conditioning velocity history \((\dot{x}_{1:T_c-1}^n, \dot{y}_{1:T_c-1}^n)\) using a sinusoidal embedding scaled by \(\gamma\), and scales the scalar visibility flags \(O_{1:T_c}^n\) by \(\beta\). Finally, the noisy target velocities \(\hat{V}\) and noisy occlusion values \(\hat{O}\) at diffusion step \(\tau\) are appended along the channel dimension to form the track token: $\(Z_n = [\mathbf{Z}_{\tau, n}^{\text{diff}}, f_n^{\text{DINO}}, V_{1:T_c}^n, O_{1:T_c}^n]\)$ Each concatenated token is projected to the transformer hidden dimension \(D_T\), followed by the addition of a 2D sinusoidal spatial positional encoding computed directly from \((x_1^n, y_1^n)\). This design allows the self-attention layers to remain permutation-invariant with respect to point order while explicitly preserving relative spatial layout and mutual non-rigid constraints among neighboring animal surface points.

3. Occlusion-Aware Velocity Reparameterization: mitigating spatial correlation and modeling visibility Directly diffusing over raw absolute coordinates \(X \in \mathbb{R}^{T \times N \times 2}\) incurs two severe training pathologies: first, absolute coordinate trajectories exhibit extreme temporal autocorrelation, where the majority of sequence variance stems from the starting position rather than incremental motion dynamics; second, frequent self-occlusions during natural locomotion (such as legs passing behind torsos) lead to missing coordinate values. To overcome these issues, the target data is reparameterized into temporal velocity increments and occlusion indicators: \(\mathbf{Z}_0^{\text{diff}} = \{\gamma V, \beta O\}\), where velocities are defined as frame differences \(\dot{x}_t^n = x_{t+1}^n - x_t^n\) and \(\dot{y}_t^n = y_{t+1}^n - y_t^n\). During occluded frames, missing values are linearly interpolated between the nearest visible endpoint frames \((i, j)\): $\(\dot{x}_t^n = \frac{x_i^n - x_j^n}{i - j}\)$ If a track remains occluded through the end of the sequence, the velocity is padded with zero. Although the visibility state \(O_t^n \in \{0, 1\}\) is nominally discrete, continuous Gaussian diffusion effectively learns to denoise to discrete extremes once properly scaled by hyperparameter \(\beta\). After multi-step denoising in the velocity space during inference, absolute trajectory coordinates are deterministically recovered via cumulative summation: $\(x_t^n = x_1^n + \sum_{s=1}^{t-1} v_{x, s}^n, \quad y_t^n = y_1^n + \sum_{s=1}^{t-1} v_{y, s}^n\)$

4. Trajectory Diffusion Transformer Denoising: permutation invariance and global displacement control To accommodate the multimodal, non-deterministic nature of natural animal movement (e.g., an animal may suddenly sprint, graze, or halt), the denoising backbone is instantiated as a Diffusion Transformer (DiT). Operating over the unordered set of \(N\) point tokens, standard multi-head self-attention naturally maintains permutation equivariance, scaling seamlessly across varied species morphology regardless of track count. Beyond per-track tokens, the network incorporates global conditioning: the diffusion timestep \(\tau\) and an optional high-level displacement vector \(d \in \mathbb{R}^2\): $\(d = \frac{\sum_{n=1}^N O_T^n \left[(x_T^n, y_T^n) - (x_1^n, y_1^n)\right]}{\sum_{n=1}^N O_T^n}\)$ These global conditioning variables are projected via linear embeddings and modulated into each transformer block via adaptive layer normalization (AdaLN). When displacement guidance is absent, the embedding of \(d\) is zeroed out to facilitate unconditional behavioral exploration; when supplied, AdaLN modulates trajectory tokens to adhere to specified directional goals while preserving biologically plausible articulation and non-rigid deformations.

Loss & Training

The forward diffusion process follows DDPM, injecting Gaussian noise into the reparameterized targets \(\mathbf{Z}_0^{\text{diff}}\): $\(q(\mathbf{Z}_\tau^{\text{diff}} \mid \mathbf{Z}_0^{\text{diff}}) = \mathcal{N}\left(\mathbf{Z}_\tau^{\text{diff}}; \sqrt{\bar{\alpha}_\tau} \mathbf{Z}_0^{\text{diff}}, (1 - \bar{\alpha}_\tau) \mathbf{I}\right)\)$ using a linear noise schedule from \(\beta_1 = 0.0001\) to \(\beta_S = 0.02\) across \(S=1000\) diffusion timesteps. The network \(f_\theta\) is trained to denoise and directly predict the clean target \(\mathbf{Z}_0^{\text{diff}}\) by minimizing the L1 loss: $\(\mathcal{L} = \mathbb{E}_{\mathbf{Z}_0^{\text{diff}}, \tau, \epsilon} \left[ \| \mathbf{Z}_0^{\text{diff}} - f_\theta(\mathbf{Z}_\tau^{\text{diff}}, \mathbf{Z}^{\text{cond}}, \tau) \|_1 \right]\)$ where \(\mathbf{Z}^{\text{cond}} = \{I, X_1, V_{1:T_c}, O_{1:T_c}, d\}\) aggregates the conditioning image features, initial positions, motion history, and global displacement.

During inference, DDIM deterministic sampling (\(\eta = 0\)) is employed to accelerate generation from 1000 training steps down to 100 sampling steps, yielding a \(10\times\) speedup with negligible fidelity loss. In default experiments, the model observes \(N_{\text{cond}}=4\) history frames (approximately 0.27 seconds) and forecasts 28 future frames (approximately 1.87 seconds) at 15 FPS.

Key Experimental Results

Main Results

Quantitative evaluations are conducted on the MammalMotion benchmark across non-learned baselines (No-Motion, Constant Velocity, Oracle Velocity), zero-shot general point forecasting (WHN), and robotic point diffusion (Track2Act). Metrics evaluate both distribution-level realism (Fréchet distance on velocities FD(V) and accelerations FD(A), variance Var(V) and Var(A), and Fréchet Video Motion Distance FVMD) and instance-level accuracy using Best-of-5 sampling (Average Displacement Error ADE, Final Displacement Error FDE, Video Motion Distance VMD, and Points Within Threshold PWT).

The following table summarizes performance on the All Data benchmark under both High Motion and Combined motion regimes:

Regime Method FD(V)↓ (×10³) FD(A)↓ (×10³) FVMD↓ (/10³) ADE↓ FDE↓ VMD↓ PWT↑ (%)
High Motion Ground Truth - - - - - - -
High Motion No motion 27.10 7.51 481.99 0.325 0.596 6.50 12.44
High Motion Constant vel 13.70 7.51 210.47 0.286 0.591 5.02 11.94
High Motion WHN (Zero-shot) 25.20 3.19 280.77 0.262 0.538 5.74 11.62
High Motion Track2Act 11.80 4.81 76.28 0.136 0.294 4.50 21.79
High Motion Ours (uncond) 8.96 3.74 49.30 0.119 0.275 4.33 26.01
High Motion Oracle vel (Reference) 12.10 7.51 326.80 0.110 0.156 7.04 14.70
High Motion Ours (cond) 4.86 3.33 40.24 0.068 0.103 4.25 31.50
Combined No motion 4.66 1.53 204.14 0.099 0.180 4.82 53.94
Combined Constant vel 2.59 1.53 89.77 0.104 0.215 5.02 41.15
Combined WHN 5.34 0.691 94.70 0.105 0.200 4.85 29.92
Combined Track2Act 2.52 1.09 26.17 0.053 0.110 3.48 56.59
Combined Ours (uncond) 1.96 0.877 17.00 0.046 0.102 3.31 60.01
Combined Oracle vel (Reference) 2.26 1.53 185.62 0.042 0.058 5.57 51.74
Combined Ours (cond) 1.07 0.778 14.38 0.028 0.042 3.26 63.48

Ablation Study

To assess the impact of cross-species pretraining versus domain-specific training on a single genus, as well as deterministic coordinate regression (ATM) versus diffusion modeling, experiments were conducted on the Panthera genus subset:

Training Setup / Method Evaluation Regime FD(V)↓ (×10³) FD(A)↓ (×10³) FVMD↓ (/10³) ADE↓ FDE↓ PWT↑ (%) Note / Key Insight
No motion Panthera (High Motion) 16.60 5.61 335.41 0.211 0.393 13.22 Fails under significant dynamic motion
Constant vel Panthera (High Motion) 7.49 5.61 149.52 0.193 0.413 16.73 Rigid extrapolation causes unnatural drift
ATM (Regression) Panthera (High Motion) 6.52 6.18 112.71 0.143 0.262 16.31 Lacks distribution diversity; blurry mean output
Track2Act (Trained on Panthera) Panthera (High Motion) 6.32 4.30 104.85 0.135 0.245 16.72 Diffuses directly on absolute coordinates
Ours (Trained on Panthera) Panthera (High Motion) 3.71 5.06 84.79 0.107 0.209 20.68 Strong within-domain specialized performance
Ours (Pretrained on All Data) Panthera (High Motion) 1.96 3.74 49.30 0.119 0.275 26.01 Demonstrates massive positive cross-species transfer
Ours (Displacement-conditioned) Panthera (High Motion) 2.82 4.19 79.38 0.067 0.097 27.31 Guiding displacement halves endpoint error

Key Findings

  • Positive cross-species transfer from large-scale pretraining: Training on the broad diversity of the 300-hour MammalMotion dataset substantially outperforms training exclusively on the specialized Panthera subset, cutting FVMD on high-motion Panthera instances from 84.79 to 49.30 and raising PWT from 20.68% to 26.01%. This highlights that non-rigid animal kinematic patterns share universal biomechanical structures across species.
  • Generative diffusion versus deterministic regression: Regression-based tracking models such as ATM suffer from high acceleration distribution error FD(A) (6.18 vs 3.74/5.06) and elevated motion discrepancy FVMD. ATM outputs an over-smoothed statistical average trajectory, whereas diffusion modeling captures multimodal, biologically authentic movement dynamics and phase variations.
  • Controllable global guidance without compromising local articulation: Providing the average displacement vector \(d\) dramatically cuts displacement error (reducing High Motion ADE from 0.119 to 0.068 and FDE from 0.275 to 0.103). Crucially, while a naive Oracle Velocity baseline achieves low displacement error, its FVMD remains poor (326.80), demonstrating that high-level direction guidance combined with DiT token attention successfully decouples macro-displacement from micro-articulation.
  • Robustness on long-tail categories over video diffusion (SVD): On long-tail species like hares (which constitute only 0.39% of the training distribution), pixel-level video diffusion models (Stable Video Diffusion) struggle and exhibit structural hallucinations, morphing hare ears into avian wings. In contrast, by operating on point trajectories, the proposed model produces physically consistent, grounded foraging and head-turning motions.

Highlights & Insights

  • Representation shift from pixels to trajectory tokens: Abstracting dynamic behavior into dense point trajectories bypasses the massive compute and data demands of pixel-level video synthesis. This mid-level tokenization allows the diffusion transformer to focus exclusively on articulation and non-rigid physical laws.
  • Exploiting telephoto wildlife optics for camera compensation: Leveraging the optical characteristics of wildlife videography (distant perspectives with minimal perspective distortion), the system employs RANSAC homographies over static background tracks to cancel camera motion, establishing an effective world-coordinate surrogate without explicit 3D geometry.
  • Remarkable zero-shot generalization to non-mammal and synthetic agents: Despite being trained primarily on mammal footage, the model demonstrates striking zero-shot generalization to humans, birds (ostriches), insects (butterflies), and even rigid/non-rigid synthetic objects (Lego robots), generating biomechanically consistent future trajectories.

Limitations & Future Work

  • Ambiguities under full 3D body rotation: Because the model operates on 2D projected coordinates, out-of-plane 3D rotations, somersaults, or severe occlusions caused by dense foliage can induce cumulative trajectory drift that 2D interpolation cannot fully correct.
  • Sensitivity to parallax in near-field video: The homography-based camera stabilization relies on distant viewpoints. In close-up drone footage or complex multi-plane scenes with prominent foreground parallax, residual apparent motion can contaminate the world-coordinate approximation.
  • Visualization reliance on patch re-projection: Currently, predicted trajectories are visualized by translating circular patches from the initial observation frame. Integrating this trajectory prediction pipeline with trajectory-conditioned controllable video synthesis remains an exciting open avenue.
  • vs Track2Act / Any-Point Trajectory Modeling (ATM): Robotic point prediction methods generally assume rigid manipulation scenes with fully visible markers, diffusing directly over absolute coordinates. The proposed method handles complex, non-rigid animal anatomy by introducing velocity reparameterization, local DINOv3 semantic tokens, and explicit visibility modeling.
  • vs Video Diffusion Models (SVD / Sora): Generative video models struggle with physical commonsense and long-tail species due to the burden of synthesizing textures, lighting, and background details. Point trajectory forecasting achieves higher physical realism and data efficiency with substantially lower computational overhead.
  • vs Parametric 3D Animal Reconstruction (MagicPony / 3D Safari / BITE): Prior 3D methods rely heavily on species-specific parametric mesh templates (zebras, horses, dogs). In contrast, the point token formulation is category-agnostic and generalizes across arbitrary morphologies.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates dense point tracks as behavior tokens, uncovering log-normal animal motion statistics and proposing an occlusion-aware DiT framework.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Curates the 300-hour MammalMotion benchmark with exhaustive distribution-level, instance-level, and cross-species ablation analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear conceptual motivation, cohesive narrative flow, and precise mathematical and algorithmic exposition.
  • Value: ⭐⭐⭐⭐⭐ Bridges computer vision, robotics, and computational ethology, offering a scalable tokenized paradigm for behavior forecasting.