Bootstrapping Articulated 3D Reconstruction from 2D Image Collections¶

Conference: ECCV2026
Paper: ECCV Official Page
PDF: Full Paper
Project: BAT3R
Authors: Jakub Zadrozny, Oisin Mac Aodha, Hakan Bilen
Area: 3D Vision
Keywords: articulated reconstruction, weak supervision, dual point maps, mesh fitting, synthetic training data

TL;DR¶

BAT3R initializes a 3D predictor from a single canonical-pose mesh, then fits articulations from 2D images, re-renders them, and trains iteratively, reducing horse RMS Chamfer Distance on Animodel-Points from 7.88 cm to 5.65 cm, while remaining above the 4.30 cm achieved by fully 3D-supervised DualPM.

Background & Motivation¶

A single photograph exposes only part of an animal's surface, while bent legs, occluded limbs, and camera orientation jointly affect its projection. Reconstruction must therefore recover geometry while separating articulation from viewpoint. DualPM represents the same surface through paired posed and canonical point maps and predicts both visible and occluded regions, but its training relies on existing articulated 3D assets. Creating natural joint motion for categories such as horses and cows is much harder than sampling cameras; additional views of a stationary mesh cannot supply the missing articulation diversity.

Meanwhile, online photographs of a category already contain many natural poses, but lack paired 3D annotations. Treating a weak predictor's point clouds as ground truth would feed limb confusion, local noise, and implausible deformation back into the model. The paper takes a narrower route: instead of recovering exact ground truth for each photograph, extract a useful articulation, apply it to a structurally credible mesh, and manufacture new supervision in which images and geometry are precisely paired.

The method is consequently not free of 3D priors. Each category still needs a rigged and skinned canonical mesh, while the image collection needs foreground masks and a coarse camera-viewpoint prior. Core Idea: use noisy point clouds to suggest articulation, use the template mesh to preserve the structure of training geometry, and turn the two into self-consistent supervision through re-rendering and iterative predictor improvement.

Method¶

Overall Architecture¶

Training uses an unordered collection of RGB images and foreground masks from one category, together with an animation-ready canonical mesh. Images may show different individuals and need not have paired views, camera annotations, or 3D labels. BAT3R contributes a training-data generation procedure; the single-image reconstruction network remains DualPM rather than a new point-map architecture.

The procedure begins with “Canonical-Pose Initialization,” then repeatedly performs “Structurally Constrained Fitting” and “Re-rendering Bootstrap.” The predictor produces dual point maps from images; fitting recovers mesh articulation and global alignment, and re-rendering supplies precisely paired images and point maps for the next training round. After training, the predictor takes one image and outputs a 3D point cloud; the fitting module can additionally produce a structured mesh.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Canonical mesh and skeleton"] --> B["Canonical-Pose Initialization"]
    B --> C["Current predictor<br/>outputs dual point maps"]
    I["2D images and foreground masks"] --> C
    C --> D["Structurally Constrained Fitting"]
    D --> E["Re-rendering Bootstrap"]
    E -->|Train next-round predictor| C
    C --> O["Single-image 3D point cloud"]

Key Designs¶

1. Canonical-Pose Initialization: learn coordinate correspondences before learning motion

For each foreground pixel, DualPM predicts multiple intersections between the camera ray and the surface, rather than only the first visible point. Each intersection has two coordinates: its posed point lies in the camera frame and describes its 3D position in the photograph, while its canonical point lies in the object's rest-pose frame and identifies a surface location on the template. Both refer to the same physical surface. Even when a predicted leg position is inaccurate, the canonical point may still tell the fitter which part of the leg should be pulled, instead of confusing adjacent limbs through nearest-distance matching in posed space alone.

Initialization renders the single rest-pose mesh from different viewpoints, with texture and environment lighting, to create exact image–dual-point-map pairs. This does not produce the true articulation distribution, but provides coarse shape, viewpoint, and canonical correspondences sufficient to initialize fitting. Camera sampling is not an incidental augmentation: the ablation replaces Gaussian sampling with a heavy-tailed Student’s t distribution to expose initialization to more extreme viewpoints. If the initial model is familiar only with lateral views, its viewpoint errors can constrain the subsequent loop.

2. Structurally Constrained Fitting: extract articulation from imperfect point clouds rather than copy their geometry

For each predicted point pair, the canonical point is assigned to its nearest vertex on the canonical mesh, and the paired posed point becomes that vertex's fitting target in camera space. Real individuals differ anatomically from the single template, and predictions are noisy, so fitting every point can overfit. The method retains only 200 control points using topology-aware farthest point sampling: nearby points in 3D space may belong to different limbs, so a small Euclidean distance should not cause one side to be discarded.

The sampling distance adds the connection from a predicted point to the template surface, the geodesic path along that surface, and the connection from the surface to the other predicted point. Restating equation (2) using its accompanying prose gives:

\[ D(q_n,q_m)=\|q_n-v^c_{j^*(n)}\|_2+d_{\mathrm{geo}}(v^c_{j^*(n)},v^c_{j^*(m)})+\|v^c_{j^*(m)}-q_m\|_2. \]

Here, \(q_n,q_m\) are predicted canonical points, \(v^c_{j^*(n)},v^c_{j^*(m)}\) are their respective nearest canonical mesh vertices, and \(d_{\mathrm{geo}}\) is geodesic distance along the mesh surface. Control points can thus cover the surface more evenly rather than being misled by limbs crowded together in space.

Fitting starts with a Kabsch-based global alignment step handling scale, rotation, and translation, then jointly optimizes joint rotations relative to the rest pose, scalar per-bone scales, and the global transform. A Huber loss with threshold 0.1 reduces the influence of outliers in point-to-corresponding-vertex distances. Joint-angle penalties and local volume and edge-length constraints discourage arbitrary bending or stretching; bone-scale constraints enforce left–right symmetry and suppress extreme proportions. A self-repulsion term penalizes vertices that are far apart on the surface but collapse together in articulated space. Together, these terms change the objective from minimizing error at any cost to following the articulation within structural constraints.

An important detail is that bone scales only absorb anatomical differences between real individuals. Volume, edge-length, and self-repulsion penalties are computed on the articulated mesh without these bone scales, avoiding conflicts between structural regularization and valid size variation. The scales are also discarded after fitting, and the unscaled articulated mesh generates supervision for the next round. For synthetic training images without such individual variation, the experiments disable bone-scale optimization altogether.

3. Re-rendering Bootstrap: align supervision with new images instead of forcing it onto old photographs

The fitted mesh may not be an exact 3D explanation of the original photograph, but it retains template structure while incorporating articulation extracted from that image. BAT3R does not present this approximate mesh as ground truth for the original image. Instead, it re-renders the mesh with recovered camera parameters, producing a new image and its genuinely corresponding posed and canonical point maps. “Exact supervision” refers only to geometric consistency within these synthetic pairs, not to error-free articulation estimates for the original photographs.

The method also renders fitted meshes from additional novel viewpoints, addressing the imbalance between easier lateral-view fitting and harder frontal or rear views. The next predictor trains on these new synthetic pairs and returns to the original image collection to extract better articulations. The loop combines pose diversity from real images with structural priors from the template, rather than restricting learning to the initial rest pose or repeatedly learning from the same unconstrained noisy point clouds.

A Worked Example¶

Consider a side-view photograph of a horse lifting a leg. This illustrates the procedure rather than reporting an additional experiment. The initial predictor may recover the coarse torso and camera direction while predicting the raised leg as too straight; canonical points can still provide correspondences between that leg and the template.

The fitter selects 200 control points from the dual point clouds, carries canonical correspondences into posed space, and brings the target points closer through joint rotation while geometric constraints discourage local stretching and limb intersections. If the actual horse's leg length differs from the template, auxiliary bone scales first absorb that discrepancy. Re-rendering then discards those scales while retaining the extracted joint articulation.

The system renders the articulated mesh and its novel views, giving naturally paired images and dual point maps. The next model has therefore seen a training example closer to a raised-leg articulation than the rest pose, without assuming that the point cloud predicted for the original photograph was already correct ground truth.

Loss & Training¶

Point-map prediction retains DualPM's posed-point and canonical-point regression objectives, summed together and weighted using predicted confidence. Mesh fitting is a separate gradient-based optimization with the robust data term and geometric regularizers described above; the entire rendering loop is not trained in one pass through a single end-to-end loss.

The main paper explains that using the original DualPM training procedure directly becomes unstable because early fitted meshes contain extreme articulation outliers, especially with real images. It therefore adopts an optimization scheme tailored to noisy targets. Details are deferred to the supplement; the local cache contains only the main paper and has damaged extraction of some equations, so this note does not invent learning rates, regularization weights, or the complete confidence loss. The text says refinement typically runs for four rounds, whereas the iteration analysis in Figure 5 extends to the seventh round. Not every reported result should therefore be assumed to come from four rounds.

Key Experimental Results¶

Main Results¶

The horse, cow, and sheep experiments on Animodel-Points use DualPM's synthetic-image splits, but BAT3R does not access their ground-truth articulations or camera parameters. The table below selects central values from the paper's Table 1 and omits its dispersion terms. All values are in cm, and lower is better. RMS CD is the RMS measure of bidirectional Chamfer Distance, evaluated with ICP alignment that includes rotation. MV RMS CD permits only scale and translation alignment, so it also penalizes camera-rotation errors.

Category	Metric ↓	Canonical	BAT3R (synt)	DualPM (full 3D supervision)
Horse	RMS CD	7.88	5.65	4.30
Cow	RMS CD	6.66	4.17	3.18
Sheep	RMS CD	6.62	4.35	3.30
Horse	MV RMS CD	9.87	7.73	5.49
Cow	MV RMS CD	7.85	5.30	4.03
Sheep	MV RMS CD	7.70	5.04	4.22

Horse-Robust tests template transfer on 15,000 samples: the male horse template is available during training, while evaluation uses the female template. BAT3R (real) extracts articulations from approximately 11,000 real horse images from MagicPony, but still requires a canonical mesh and synthetic rendering supervision; it does not eliminate 3D data entirely. The paper's Table 2 reports:

Method	RMS CD (cm) ↓	MV RMS CD (cm) ↓
Trellis.2	8.92	40.03
SAM-3D	6.42	8.71
DualPM	3.84	5.42
Canonical	7.84	10.83
BAT3R (synt)	5.10	7.92
BAT3R (real)	6.01	8.37

This is not a ranking under identical conditions: BAT3R is category-specific, while the two foundation models are evaluated zero-shot. Trellis.2 additionally relies on best-effort post-hoc rotation alignment, whose failures inflate its error. Real chimpanzee and elephant results are qualitative only and do not establish quantitative superiority across categories.

Ablation Study¶

The paper's Table 3 uses Horse-Mixed: 18,000 in-distribution test samples, with both male and female templates present during training and evaluation. Its absolute errors should not be directly mixed with Horse-Robust results. This is a cumulative addition of components, not an independent removal of each component. mCD is mean Chamfer Distance; the outlier ratio is the percentage of samples with error above 5 cm; \(\Delta R\) is angular camera-rotation error. Lower is better for all three.

Config	mCD (cm) ↓	Outlier ratio (%) ↓	\(\Delta R\) (°) ↓
Canonical (Gaussian camera sampling)	10.04	92.0	12.23
+ Student’s t camera sampling	7.24	68.4	7.51
+ Refinement without fitting regularization	5.59	55.1	6.73
+ Mesh-fitting regularization	4.85	35.0	5.89
+ Novel-view augmentation, full method	3.86	11.3	3.49
Random poses baseline	5.22	44.5	5.12
DualPM (ground-truth 3D supervision)	3.04	4.00	2.21

Key Findings¶

On Animodel-Points, BAT3R's horse RMS CD is 2.23 cm below Canonical but 1.35 cm above DualPM. The supported conclusion is a narrower supervision gap, not superiority to full supervision.
In the cumulative ablation, fitting regularization reduces mCD from 5.59 to 4.85 cm; adding viewpoint augmentation reduces it further to 3.86 cm. The outlier ratio falls from 35.0% to 11.3%, a reduction of 23.7 percentage points. Viewpoint coverage improves both average error and the incidence of severe failures.
The random poses baseline reaches 5.22 cm, showing that pose diversity alone is useful training signal. The full method's 3.86 cm demonstrates further gains from structural constraints and image-driven articulation extraction. The iteration analysis shows saturation, not evidence that unlimited iterations would keep delivering large improvements.

Highlights & Insights¶

The key distinction is between accuracy on the original image and consistency of the new training pair. Re-rendering can ensure the latter without requiring perfection in the former, avoiding direct treatment of prediction noise as image ground truth.
Dual point maps provide more than a complete shape output: they supply semantic correspondences for fitting. Canonical-space matching and surface-distance sampling help a limited set of control points constrain different limbs.
Bone scales are auxiliary fitting variables rather than obligatory components of the target shape. Absorbing individual variation and then discarding it helps separate articulation learning from changes in template proportions.

Limitations & Future Work¶

The authors acknowledge the continuing need for a rigged and skinned canonical mesh and the performance gaps among real-image training, synthetic-image training, and full 3D supervision. Foreground masks, texture, and a coarse viewpoint prior are also practical input requirements.
The template limits representable structure. Geometric regularization suppresses deformation artifacts but does not guarantee recovery of the original image's true articulation. Discarding bone scales supports standardized supervision but is not a complete solution to detailed individual shape reconstruction.
The current cache lacks the supplementary training details, preventing verification of all optimization hyperparameters and computational costs. Evidence for chimpanzees and elephants remains primarily visual, so generalization claims should stay within the demonstrated scope.

vs DualPM: BAT3R retains its dual-point-map representation and predictor while replacing the acquisition of paired articulated training data. It substantially reduces 3D asset requirements, but its quantitative accuracy remains below DualPM trained with ground-truth 3D articulations.
vs MagicPony / 3D-Fauna: These methods learn articulated shapes from 2D collections, whereas BAT3R explicitly introduces an existing rigged mesh and regenerates paired supervision. Comparisons must account for both accuracy and prior requirements rather than treating it as a comprehensive replacement under equal supervision.
vs SAM-3D / Trellis.2: Foundation models offer cross-category zero-shot reconstruction; BAT3R uses category templates and dedicated training to handle complex articulation. Horse-Robust numbers support an advantage in this setting, not stronger open-category reconstruction capability.

Rating¶

Novelty: 4/5. The main contribution is structurally constrained data bootstrapping rather than a new reconstruction representation.
Experimental Thoroughness: 4/5. Synthetic and real-image training and cumulative ablations are covered, but additional animal categories lack quantitative validation.
Writing Quality: 4/5. The central argument is clear, but important implementation details depend on the supplement, and protocols and iteration settings require careful distinction.
Value: 4/5. It offers a practical training route for categories with scarce 3D assets, while retaining template requirements and an accuracy gap.