BioMTBee: Biologically Constrained Multi-View Template-Based 3D Reconstruction of Bumblebee¶

Conference: ECCV 2026
Paper: Official paper page · Paper PDF
Area: 3D Vision
Keywords: Animal 3D reconstruction, multi-view pose estimation, articulated template, biological constraints, differentiable rendering

TL;DR¶

BioMTBee fits a micro-CT-derived bumblebee template to multi-view video using filtered 3D poses, silhouettes, and biological constraints, reducing mesh-keypoint error from MAMMAL's 0.189 mm to 0.152 mm while increasing silhouette IoU from 0.696 to 0.758 on the DBS test split.

Background & Motivation¶

Animal keypoints describe where limbs move, but not the continuous surface, appendage thickness, or individual morphology needed for detailed motion analysis. Bumblebees are difficult subjects because dark dense hair weakens visual cues, slender limbs and antennae frequently occlude one another, and rapid motion produces missed or erroneous 2D detections. Template-free reconstruction lacks strong anatomical guidance where observations are ambiguous, while generic animal models such as SMAL mainly target vertebrates rather than insect-specific skeletons and geometry. Obtaining dense ground-truth surfaces of moving insects is also difficult, so simply adding more mesh supervision is not an immediate solution.

This paper takes a species-specific fitting approach rather than predicting a generic animal from an arbitrary image. Micro-CT supplies an offline anatomical template; synchronized multi-view video supplies the dynamic observations, making this an animal 3D reconstruction paper rather than a clinical imaging method. The template specifies plausible structure, pose and silhouette cues locate that structure in the recording, and biological constraints restrict deformations that weak visual evidence would otherwise permit. Core idea: connect a CT-derived anatomical template, robust spatiotemporal pose estimation, and biologically constrained fitting so that dynamic surface recovery respects both observations and species structure.

Method¶

Overall Architecture¶

The inputs are synchronized multi-view bumblebee videos and a previously constructed articulated CT template; the output is a time-varying mesh with per-vertex color. The pipeline prepares the CT articulated template, obtains camera geometry and stable supervision through BPST and silhouette supervision, and exposes individual deformation parameters for fitting. Biologically constrained staged fitting then uses differentiable rendering to align the mesh with observations while limiting implausible articulation. The template is an offline prior, not a scan acquired for every video frame; the two input branches represent different data sources.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Offline CT specimen"] --> B["CT articulated template"]
    V["Synchronized multi-view video"] --> C["BPST and silhouette<br/>supervision"]
    B --> D["Individual deformation<br/>parameterization"]
    C --> D
    D --> E["Biologically constrained<br/>staged fitting"]
    E --> F["Dynamic 3D mesh<br/>and vertex colors"]

Key Designs¶

1. CT articulated template: establish geometry and joint structure first

A micro-CT scan of a bumblebee specimen is converted into a surface using volumetric processing and Marching Cubes, followed by cleanup, simplification, and retopology. The resulting template contains 4,070 vertices and 8,136 faces, retaining major anatomical structures rather than explicitly reconstructing individual hairs. A thorax-rooted kinematic tree contains 63 control bones, and linear blend skinning binds vertices to those bones so articulation moves the corresponding surface coherently. A further mapping connects mesh vertices to 41 anatomical keypoints, allowing sparse pose observations to supervise the dense mesh.

The benefit is a constrained, interpretable search space: an occluded leg retains its connectivity, and the abdomen cannot deform arbitrarily merely to match a local image region. Compared with a generic statistical animal model, the main distinction is the source and species specificity of the prior, not a new skinning algorithm. The CT acquisition precision should not be confused with reconstruction accuracy in video; the simplified template and downstream fitting must still be evaluated experimentally.

2. BPST and silhouette supervision: stabilize motion evidence before surface fitting

DeepLabCut predicts 2D anatomical keypoints in each view, and observations aggregated across frames support automatic multi-view calibration. The described incremental SfM procedure starts with stereo initialization, incorporates additional views using RANSAC–PnP, rejects outliers, and refines camera extrinsics with bundle adjustment before triangulation. This addresses sensitivity to small camera shifts in a millimeter-scale setup; automatic calibration does not remove the need for multi-view geometry.

BPST refines the recovered pose using robust reprojection error, temporal smoothness, and segment-length consistency. These terms address different problems: outlier detections, frame-to-frame jitter, and implausible changes in limb length during motion. In parallel, 2D keypoints prompt HQ-SAM to extract foreground silhouettes for each view. The stable 3D points constrain articulation, whereas the silhouettes constrain projected body width and boundaries that sparse keypoints cannot specify; these masks are not dense 3D ground truth.

3. Individual deformation parameterization: separate morphology, motion, and appearance

One scanned specimen cannot represent all individuals, so per-bone anisotropic scaling along three local axes adapts the morphology of the associated mesh regions. Per-frame relative bone rotations, represented in axis–angle form, account for articulation through the hierarchical skeleton. Additional variables describe global scale, rotation, translation, and per-vertex RGB appearance. Linear blend skinning combines the bone transformations from bind pose to current pose according to vertex weights, rather than treating every vertex as an independently movable point.

This representation gives different explanations to a thicker leg and a rotated leg. Shape parameters describe comparatively stable individual differences, pose parameters explain motion, and vertex colors explain visible appearance, reducing opportunities for one variable group to compensate for another. The flexibility remains bounded by the template topology and skinning assignments: it cannot invent additional appendages or explicitly recover fine hair absent from the representation.

4. Biologically constrained staged fitting: do not sacrifice anatomy to image agreement

The data objective aligns mesh-derived keypoints with BPST's 3D estimates, rendered silhouettes with HQ-SAM masks, and rendered colors with the input images. Keypoint supervision accounts for visibility, while photometric error is weighted by the rendered foreground mask rather than requiring the animal model to explain the background. These terms constrain articulation, projected surface shape, and appearance respectively; a small keypoint error alone cannot establish correct limb thickness or abdomen shape.

Biological priors encourage matching shape scales for corresponding left–right segments, penalize joint angles outside species-specific anatomical ranges, and regularize changes in pose across adjacent frames. Bilateral symmetry concerns morphology, not a requirement that both legs execute mirrored movements simultaneously. The main text concretely implements the kinematic prior as joint-range penalties, so it should not be expanded into an unverified dynamical coupling model. BPST smooths estimated 3D keypoints upstream, whereas the reconstruction stage smooths skeletal pose variables; these operate on different representations. Together, the priors reject some solutions that project plausibly into partially occluded images but contain implausible anatomy.

Loss & Training¶

The reconstruction is a two-stage optimization of the observed sequence, not a generic feed-forward network trained to output a mesh in one pass. Stage I uses a subset of frames to optimize individual shape, global scale and rigid transformation, and vertex appearance without temporal terms. Stage II fixes shape and global parameters and optimizes per-frame skeletal pose, assigning temporal movement primarily to the articulation variables. The implementation uses PyTorch/PyTorch3D, Adam, and cosine learning-rate decay; the DeepLabCut detector is separately trained from annotated images. Consequently, reconstruction involves detection, segmentation, and sequence-specific parameter optimization, and camera capture rate is not evidence of real-time inference speed.

Evidence boundary: equations (1)–(7) contain extraction damage affecting operators, indices, or delimiters in the cached text. This note explains the objectives from their textual definitions instead of reconstructing unverifiable full equations. The exact BPST robust function and deformation derivation point to supplementary material unavailable in the local cache; loss weights and iteration budgets therefore cannot be claimed to be fully reproducible here.

Key Experimental Results¶

Main Results¶

DBS contains one bumblebee in a continuous 2,000-frame recording captured from six synchronized views at 200 FPS and 1000×800 resolution. Each view has 1,800 annotated frames: 1,500 train DeepLabCut, and the remaining 300 provide triangulated 3D keypoint ground truth (§4.1). This is a split within one individual and one continuous recording, not a cross-individual test; the main text does not fully specify temporal sampling. MPJPE measures mean 3D joint position error; PA-MPJPE measures position error after Procrustes alignment, with lower values preferred for both. 3D-PCK measures the percentage of joints within the error threshold, set to 0.35 mm for bumblebees; silhouette IoU measures intersection over union of rendered and reference masks, with higher values preferred for both.

The following selected comparison is from Table 3 on DBS; mesh methods obtain pose metrics by mapping vertices to keypoints.

Method	MPJPE ↓ (mm)	PA-MPJPE ↓ (mm)	[email protected] mm ↑ (%)	Silhouette IoU ↑
DANNCE	0.229	0.217	84.9	Not reported
DANNCE-T	0.219	0.206	87.3	Not reported
MAMMAL	0.189	0.178	92.0	0.696
BioMTBee	0.152	0.147	96.6	0.758

Against MAMMAL, MPJPE decreases by 0.037 mm, approximately 19.6%; PCK increases by 4.6 percentage points and IoU by 0.062. The authors state that methods share the same split, but DANNCE variants only report pose metrics, so missing IoU values cannot be interpreted as poor mesh performance. Nor does this establish equal priors or compute: BioMTBee uses a dedicated CT template, and the main text does not fully document baseline adaptation or resource budgets.

Ablation Study¶

The following selected ablation comes from Table 2 under the bumblebee mesh evaluation protocol, not the upstream BPST experiment.

Configuration	MPJPE ↓ (mm)	PA-MPJPE ↓ (mm)	[email protected] mm ↑ (%)	Silhouette IoU ↑
Without symmetry constraint	0.163	0.156	93.6	0.724
Without kinematic constraint	0.166	0.160	93.1	0.717
Without silhouette loss	0.148	0.144	96.9	0.652
Full method	0.152	0.147	96.6	0.758

Removing the kinematic constraint causes a larger pose degradation than removing symmetry, but removing silhouette loss causes the clearest shape degradation: IoU falls from 0.758 to 0.652. Importantly, its MPJPE improves from 0.152 to 0.148 mm, demonstrating that better sparse-point alignment can coexist with worse surface projection. The prose following Table 2 incorrectly describes this as a 0.004 mm MPJPE increase; this note follows the tabulated values rather than repeating that contradictory statement.

Key Findings¶

Upstream BPST and final mesh accuracy are different: Table 1 reports BPST MPJPE of 0.110 mm, whereas the final mesh reaches 0.152 mm. The former should not be presented as mesh accuracy.
BPST MPJPE becomes 0.125 mm without robust reprojection handling, 0.119 mm without the temporal term, and 0.115 mm without segment-length consistency. Outlier handling has the largest effect among these removals (Table 1).
Biological validity is measured separately: the FTi joint violation rate beyond anatomical limits by more than 10° falls from 43.3% without the kinematic constraint to 7.9% with the full method (Table 4).
On mice, MPJPE is 2.01 mm and IoU is 0.774, versus MAMMAL's 2.43 mm and 0.751 (Table 6). This uses a mouse-specific mesh and anatomical limits, supporting framework adaptation rather than zero-shot transfer of the bumblebee template.

Highlights & Insights¶

Use accurate scanning as a prior, not as the dynamic input: offline CT addresses missing insect templates while multi-view images drive motion recovery. Static anatomical knowledge compensates for incomplete dynamic observations.
Clean the skeleton before fitting the surface: unfiltered detection errors could otherwise become implausible mesh deformations. BPST reduces outliers and spatiotemporal inconsistency before they guide reconstruction.
Check whether accurate points conceal inaccurate surfaces: the silhouette ablation moves pose and shape metrics in opposite directions. Animal mesh evaluation benefits from separate pose, surface, and anatomical checks.

Limitations & Future Work¶

Author-proposed future work: expand the dataset and insect species coverage, and improve automation and generalization in real biological settings (§6). The main text has no dedicated limitations section.
Reader assessment—limited generalization evidence: DBS covers one individual in one controlled recording, leaving robustness to body variation, capture conditions, and outdoor occlusion unresolved; mouse transfer requires a new template and biological priors.
Reader assessment—limited surface ground truth: 3D keypoints are triangulated from annotations, and shape is mainly evaluated through silhouettes. This is not independent measurement of the complete moving 3D surface.
Reader assessment—priors can introduce bias: bilateral symmetry and joint ranges suit typical anatomy but could suppress genuine asymmetry or abnormal motion. Such cases are not systematically tested.
Reproduction and deployment boundary: no supplementary material or verified project code link was available locally, and the main text does not provide runtime or a complete optimization budget. Capture speed must not be reported as reconstruction throughput.

Versus SMAL / AniMer: statistical or animal-family models emphasize generalization within existing shape spaces; BioMTBee trades broader applicability for a dedicated CT prior with insect-specific appendages and topology.
Versus MAMMAL: both use templates for dynamic animal surface recovery. The relevant additions are the bumblebee anatomy, robust multi-view pose supervision, and biological constraints—not simply having a template rather than none.
Versus DANNCE / DANNCE-T: these provide 3D animal pose references, whereas BioMTBee also reconstructs a renderable surface and appearance. Pose comparisons are informative, but their output tasks are not identical.
Versus template-free approaches: Pad3r, fauna, and SAM 3D mainly provide qualitative comparisons here. Selected examples do not establish that these methods underperform for all animals or observation settings.

Rating¶

Novelty: 3.5/5 — A targeted integration of bumblebee anatomy, robust pose estimation, and biological constraints, built on established skinning and optimization tools.
Experimental Thoroughness: 3/5 — Pose, mesh, anatomical validity, and mouse adaptation are evaluated, but single-individual data and limited reproduction details constrain the conclusions.
Writing Quality: 3/5 — The pipeline is understandable, but the silhouette-ablation prose contains a direction error, and equation extraction damage impedes local verification.
Value: 4/5 — A useful reference for detailed insect reconstruction, particularly its demonstration that sparse pose and surface quality require joint evaluation.