Progressive Pose-Guided 4D Animal Reconstruction from Monocular Video

Conference: ECCV2026
arXiv: 2607.00157
Code: None
Area: 3D Vision
Keywords: 4D Reconstruction, Animal Reconstruction, 3D Gaussian Splatting, Pose Estimation, Progressive Optimization

TL;DR

This paper proposes a progressive test-time optimization framework based on 3D Gaussian Splatting. By decoupling skeletal motion from non-rigid deformation, it reconstructs high-fidelity 4D animal models from monocular videos, requiring only coarse-grained shape priors to generalize to diverse species.

Background & Motivation

Reconstructing 4D animal models from monocular videos holds significant value for wildlife monitoring, animal behavior studies, and immersive content creation. Unlike human reconstruction—which is well-supported by mature parametric models like SMPL and massive 3D motion capture datasets—morphological and motion variations across animal species are extremely diverse, while annotated data is severely scarce. This challenges existing methods and traps them in a dilemma: category-specific prior-based methods (e.g., MagicPony, BARC) achieve high reconstruction quality on specific species but fail drastically when generalizing to unseen species; category-agnostic methods (e.g., FAUNA, SAOR) expand species coverage but often yield unsatisfactory reconstruction fidelity. When extended to 4D dynamic reconstruction, this dilemma is further exacerbated—mesh-based methods are limited by fixed topology, neural implicit methods suffer from high computational costs, and 3DGS-based methods (e.g., GART) still rely on parametric templates, while generative methods (e.g., GVFDiffusion, DreamMesh4D) sacrifice faithfulness to the input identity.

A common implicit assumption behind existing methods is that the shape prior must be accurate enough to support subsequent dynamic reconstruction. However, on animal datasets, obtaining accurate priors is inherently difficult—SMAL is learned from toy models, and FAUNA is learned from internet images, both offering only coarse approximations. The core insight of this paper contradicts this assumption: the shape prior does not need to be highly accurate; as long as structured decoupling allows pose estimation and deformation correction to perform their respective duties during progressive optimization, a coarse prior is sufficient to drive high-quality 4D reconstruction. Based on this, this paper proposes a progressive test-time optimization framework based on 3DGS. The reconstruction process is divided into two stages: symmetry-aware pose refinement and part-conditioned deformation. A learnable part anchor is designed as a shared identity space between the two stages to achieve full-pipeline reconstruction from monocular videos of high-fidelity 4D models.

Method

Overall Architecture

The proposed method takes a sequence of monocular video frames as input and outputs 4D Gaussian representations for each frame (supporting rendering from arbitrary viewpoints and timesteps). The overall pipeline is divided into three stages: First, canonical 3D Gaussians and a learnable skinning field are initialized from the Fauna prior, and bilateral symmetry augmentation is used to construct two complementary supervision groups. Then, in the pose refinement stage, learnable part anchors and a symmetry-aware temporal encoding are employed to estimate joint transformations for each frame, yielding posed Gaussians via Linear Blend Skinning (LBS). Finally, in the part-conditioned deformation stage, the posed Gaussians retrieve non-rigid deformation signals through cross-attention from the joint-level temporal features output by the pose stage, generating the final deformed Gaussian parameters. These two stages use different rendering supervision strategies—the pose stage only receives silhouette loss (preventing appearance gradients from contaminating poses), while the deformation stage jointly optimizes silhouette and photometric losses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Video Frame Sequence"] --> B["Fauna Prior Initialization<br/>Canonical 3D Gaussians + Learnable Skinning Field"]
    B --> C["Bilateral Symmetry Augmentation<br/>Construct Two Complementary Mirror Pairs"]
    C --> D["Pose Refinement Stage"]
    D --> D1["Part Anchors + Symmetry-Aware Temporal Encoding<br/>Self-Attention → Joint Transformations<br/>LBS → Posed Gaussians"]
    D1 --> E["Part-Conditioned Deformation Stage"]
    E --> E1["Cross-Attention: Per-Gaussian Part Embedding<br/>as Query, Joint Features as<br/>Key/Value, Outputting Deformation Offset"]
    E1 --> F["Deformed Gaussians G_{deform}"]
    F --> G["Differentiable Rendering<br/>Arbitrary Viewpoint/Timestep"]

    D1 -.->|"Joint-Level Temporal Features F_J^t<br/>as Key/Value"| E1

Key Designs

1. Bilateral Symmetry Augmentation: Exploiting Mirror Symmetry to Absorb Camera Drift

Animals typically exhibit bilateral symmetry, and horizontally flipping video frames can effectively expand viewpoint coverage. However, learning-based Fauna priors produce systematic drift bias when estimating camera parameters for flipped frames—the estimated camera of the flipped frame is not exactly equal to the result of reflecting the original frame's camera across the symmetry plane. Forcefully treating the original and flipped frames as precise mirror pairs injects this drift into optimization as conflicting supervision; treating them completely as independent samples loses the geometric constraints brought by symmetry.

The proposed solution is to construct two internally consistent symmetry groups for each viewpoint, rather than trying to use all four frames at once. Specifically, for each original frame \(I^t\) and its flipped frame \(I_{\text{flip}}^t\), the symmetry plane reflection transform \(M\) is applied to each of their camera parameters to obtain two groups: the \(\mathcal{V}_{\text{orig}}\) group containing \((I^t, C^t, P^t)\) and \((I_{\text{flip}}^t, C_{\text{sym}}^t, P_{\text{flip}}^t)\), where \(C_{\text{sym}}^t = M \cdot C^t\); and the \(\mathcal{V}_{\text{flip}}\) group containing \((I_{\text{flip}}^t, C_{\text{flip}}^t, P_{\text{flip}}^t)\) and \((I^t, C_{\text{flip\_sym}}^t, P^t)\). Within each group, exact mirror symmetry is maintained (as the reflection transform is manually applied without estimation error), while camera drift is allowed between groups. During training, one group is randomly sampled at each iteration, which both exploits symmetry to extend supervision signals and avoids conflicts caused by drift. The symmetry-aware temporal encoding is labeled with \(m\) (\(m=1\) for \(\mathcal{V}_{\text{orig}}\), \(m=-1\) for \(\mathcal{V}_{\text{flip}}\)), allowing the pose refinement stage to distinguish which camera group the input originates from, thereby learning the inter-group bias as a systematic offset rather than overfitting it as a contradiction.

2. Symmetry-Aware Pose Refinement: Part Anchors Decoupling Joint Motion

Initial poses provided by the Fauna prior are often inaccurate, especially in scenes with sparse views or pose ambiguity. The goal of the pose refinement stage is to learn the transformation parameters for each joint in each frame, mapping canonical Gaussians to posed Gaussians via LBS. Two key challenges arise: how to incorporate symmetry information while handling systematic camera group drift, and how to share stable identity representations for the same joint across different frames.

To address the first challenge, this paper designs a symmetry-aware temporal encoding: the encoding vector is constructed by concatenating two parts—\(\text{embed}(t \cdot m)\) which encodes the product of the timestep and the camera group label, leveraging the anti-symmetric property of sinusoidal encoding to structurally relate the temporal encodings of the two camera groups for the same frame; and a label \(v\) indicating whether the input is the original frame (\(1\)) or the flipped frame (\(-1\)), explicitly decoupling 2D visual symmetry from 3D camera configurations. To address the second challenge, a randomly initialized but learnable part anchor matrix \(A \in \mathbb{R}^{J \times D}\) (\(J=20\) joints, \(D=8\) dimensions) is introduced, where each anchor learns a stable identity representation of the corresponding joint. The part anchors are concatenated with the temporal encoding and passed through a self-attention module to generate joint-level temporal features \(F_J^t\), which are then projected into pose transformation parameters for each joint. A key design choice is that the pose stage is supervised solely by silhouette loss, and a stop-gradient is applied to block the backpropagation of non-pose attributes (scale, opacity, color)—ensuring that pose updates are purely driven by geometric alignment and not interfered with by texture information. The final output posed Gaussians \(G_{\text{pose}}^t\) serve as the starting point for the deformation stage.

3. Part-Conditioned Deformation: Shared Anchors Bridging Two Stages

LBS-driven pose refinement can only capture skeletal-level rigid body motions, whereas real animals exhibit non-rigid effects such as fur fluttering, muscle twitching, and toe bending. The input to the deformation stage is the posed Gaussians \(G_{\text{pose}}^t\), and the goal is to predict fine-grained geometric offsets on top of them.

The most critical design of this stage is that the deformation stage reuses the part anchors \(A\) from the pose refinement stage, but in a different manner. The skinning weights \(w_i\) of each Gaussian are inner-producted with the anchor matrix to obtain a part-aware embedding \(z_i = w_i^T A \in \mathbb{R}^D\). This embedding encodes the membership degree of the Gaussian to each joint, serving as a complementary signal to spatial position—when different body parts overlap in pose space (e.g., front and hind legs crossing), spatial coordinates alone cannot distinguish them, whereas \(z_i\) provides position-independent identity cues. The cross-attention in the deformation stage takes each Gaussian's \(z_i\) and posed geometric attributes \((x_p, q_p, s)\) as Queries, and the joint-level temporal features \(F_J^t\) output by the pose stage as Keys and Values. Through this attention mechanism, each Gaussian can selectively extract the required deformation signal from the time-varying features of its corresponding joint—front leg Gaussians refer more to the deformation patterns of front leg joints, while torso Gaussians refer more to those of torso joints. Thus, the part anchors assume two roles simultaneously: they act as the generation source of joint-level temporal embeddings in the pose stage, and as the calculation basis for per-Gaussian identity embeddings in the deformation stage, serving as a seamlessly shared implicit identity space between the two stages. The deformation stage only predicts offsets for geometric attributes \((x_d, q_d, s_d)\), while appearance attributes \((\alpha, c)\) are shared from the canonical Gaussians, ensuring temporal appearance consistency.

Loss & Training

The total loss consists of three terms: \(L_{\text{total}} = L_{\text{pose}} + L_{\text{deform}} + L_{\text{geo}}\). The pose loss \(L_{\text{pose}}\) only contains silhouette BCE + Dice loss (weight 0.2), with backpropagation of appearance attributes blocked by stop-gradient. The deformation loss \(L_{\text{deform}}\) jointly optimizes \(L_1\) + SSIM photometric loss and silhouette loss (SSIM weight 0.2, silhouette weight 0.1). The geometric regularization \(L_{\text{geo}}\) (only active during the first 10k iterations) includes three terms: Huber total variation smoothing of the skinning field (weight 10, ensuring physical plausibility of the skinning field), a part compactness loss (weight 0.1, which suppresses floating artifacts by penalizing only the smallest two eigenvalues via principal component analysis while preserving the slender structure of limbs along the primary axis), and a normal map smoothing loss (weight 0.1, which constrains local smoothness of novel-view normals using absolute cosine similarity). In terms of training strategy, the predicted poses are linearly mixed with the Fauna prior (with the mixing weight annealed from 1 to 0 over 7k iterations), and an \(L_2\) prior regularization is applied for the first 4k iterations, both of which naturally decay in the later stages of training to ensure the model is not restricted by priors. Additionally, decaying Gaussian noise is injected into the temporal coordinates in the early phase to encourage temporal smoothness.

Key Experimental Results

Main Results

Dataset	Metric	Ours	Best Baseline	Gain
DAVIS (Input Views)	PSNR / SSIM / LPIPS	26.32 / 0.920 / 0.089	D-3DGS*: 25.78 / 0.905 / 0.100	+0.54 PSNR
Online (Input Views)	PSNR / SSIM / LPIPS	26.21 / 0.923 / 0.091	DreamMesh4D: 23.86 / 0.889 / 0.131	+2.35 PSNR
APTv2 (Input Views)	PSNR / SSIM / LPIPS	24.94 / 0.878 / 0.139	DreamMesh4D: 26.00 / 0.899 / 0.102	-1.06 PSNR (Second)
DAVIS (Novel Views)	KID-16V↓ / FVD-F↓ / FVD-Diag↓	0.108 / 895.7 / 696.2	D-3DGS*: 0.199 / 1192.0 / 980.7	Best overall
Online (Novel Views)	KID-16V↓ / FVD-F↓ / FVD-Diag↓	0.160 / 972.0 / 1245.4	D-3DGS*: 0.231 / 1244.4 / 1268.1	Best overall
Artemis (Novel Views)	PSNR / SSIM / LPIPS	24.03 / 0.938 / 0.065	D-3DGS*: 20.05 / 0.903 / 0.103	+3.98 PSNR
Artemis (3D Geometry)	Chamfer↓ / F@1%↑ / F@2%↑	0.0156 / 0.439 / 0.760	D-3DGS*: 0.0181 / 0.358 / 0.683	Best overall

Ablation Study

Configuration	PSNR	SSIM	LPIPS	Note
Full model	24.03	0.938	0.065	Full model
w/o Deformation Module	22.80	0.929	0.072	Non-rigid dynamics are crowded into the canonical space, and subtle motions like toes are lost.
w/o Shape Prior	23.89	0.937	0.068	The canonical space is asymmetric, causing misaligned crossing of the front legs when the pose is flipped.
w/o Part Anchors	23.35	0.927	0.086	Only one of the two legs is reconstructed, while the other is approximated by stretched Gaussians.
w/o Symmetry Encoding	23.87	0.937	0.068	The canonical space is asymmetric, causing misalignment of the front legs.
Fixed Skinning Field	23.93	0.938	0.067	The canonical space contains only one leg, forcing the deformation stage to compensate for the missing limb.

Key Findings

• Deformation module contributes the most: Removing it leads to a 1.23 drop in PSNR, but the drop in perceptual quality is even more pronounced (LPIPS increases by 0.007). Moreover, the leakage of non-rigid dynamics into appearance in the canonical space causes geometric messiness.
• Part anchors are most critical for perceptual quality: Removing them results in a minor PSNR drop (0.68), but LPIPS rises by 0.021, and only one of the two legs is reconstructed. This indicates that part identity information is far more essential for accurate limb reconstruction than what PSNR reflects.
• Symmetry encoding and shape priors are complementary: Removing either individually leads to similar asymmetry issues in the canonical space. The symmetry encoding provides bilateral constraints, while the shape prior provides symmetric initial positions; both are indispensable.
• In sparse-frame scenarios (APTv2 15 frames): DreamMesh4D outperforms Ours in input-view PSNR due to its strong generative prior. However, Ours still significantly leads in novel-view KID-V and temporal stability (FVD), demonstrating that while generative approaches compensate for missing viewpoint information in few-frame settings, they do so at the cost of view consistency.

Highlights & Insights

• "Coarse Prior + Progressive Decoupling" design philosophy: Instead of striving for perfect initialization, the prior is allowed to evolve continuously during optimization. By decoupling skeletal motion and non-rigid deformation, two supervision paths are made to perform their respective roles. This methodology offers general inspiration for any scenario where the prior is weak but can provide stable optimization signals.
• Dual-role design of part anchors: Generating joint-level temporal features in the pose stage and providing per-Gaussian part embeddings via skinning weight aggregation in the deformation stage allows for seamless coupling between the two optimization stages without requiring additional feature alignment or transformation networks.
• Symmetry-aware temporal encoding handles systematic drift: By leveraging the anti-symmetric property of sinusoidal encoding via the \(m\) label and \(t \cdot m\) multiplication, the inter-group camera drift is converted from conflicting supervision into a learnable system bias. This is an elegant solution for handling scenarios where "the estimator itself has systematic bias."

Limitations & Future Work

• Limited robustness to severe camera instability: When both the original and flipped frame sequences are affected by unstable camera estimation, the bilateral augmentation cannot recover, and the optimization may converge to a sub-optimal solution.
• Mask quality depends on Grounded-SAM: Without manual correction, inaccurate segmentation regions degrade geometric accuracy. While self-occlusion in side views is partially mitigated by bilateral augmentation, persistent occlusion remains a limitation.
• Difficult to capture head movements: Head rotation causes subtle silhouette changes that cannot be sufficiently constrained by silhouette supervision alone. Normal map smoothing regularization mitigates head splitting but does not fully solve the issue.
• Current evaluation relies on rendering metrics as indirect proxies for geometric accuracy, lacking direct evaluation against ground-truth 3D geometry on wild videos.

Related Work & Insights

• vs GART: GART treats pose and deformation as isolated stages and relies on the D-SMAL parametric template. When the template mismatches (e.g., a sitting panda), the reconstruction fails. In contrast, this paper achieves seamless coupling of the two stages by sharing identity space via part anchors, and replaces the fixed template with a learnable skinning field, significantly expanding the range of manageable poses.
• vs D-3DGS: D-3DGS is general dynamic 3DGS, lacking animal semantic constraints and prone to floating artifacts. The skeletal-guided deformation in this paper constrains the space of Gaussian position updates, and the symmetry encoding leverages bilateral information to make the canonical space more stable.
• vs Generative Methods (GVFDiffusion/DreamMesh4D): Generative methods produce "visually plausible but unfaithful to input" results, as their canonical spaces bias toward the training set distribution rather than the specific input. This paper adheres to pure reconstruction supervision, allowing the canonical space to evolve continuously, which guarantees identity fidelity.

Rating

• Novelty: ⭐⭐⭐⭐ [The concept of "coarse prior + progressive decoupling" is pioneering in 3D animal reconstruction, and the dual-role design of bilateral symmetry augmentation and part anchors is elegant and pragmatic]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive comparison against 5 baselines across 4 datasets (including 87 wild videos + synthetic multi-view data), providing multi-dimensional quantitative evaluation of input/novel views, RGB, and 3D geometry; the ablation studies cover all components and conduct in-depth analysis on the root causes of canonical space visualization]
• Writing Quality: ⭐⭐⭐⭐ [The methodology is clearly described, the pipeline is complete, and the ablation analysis deeply explores canonical space visualization; it is slightly regrettable that some details such as normal smoothing in the supplementary materials are not publicly available]
• Value: ⭐⭐⭐⭐⭐ [Significantly enhances the generalization capability and fidelity of 4D animal reconstruction from monocular videos, directly driving practical applications such as wildlife monitoring, animal behavior studies, and virtual content creation]

Related Papers

• [ECCV 2026] One Video, One World: Turning Monocular Video into Physical 4D Scenes
• [ECCV 2026] HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration
• [NeurIPS 2025] Web-Scale Collection of Video Data for 4D Animal Reconstruction
• [ICCV 2025] Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpainting
• [CVPR 2026] Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video