Human Mesh Modeling for Anny Body

Conference: ECCV2026
Paper: ECCV Paper
Authors: Romain Brégier, Guénolé Fiche, Laura Bravo-Sánchez, Thomas Lucas, Matthieu Armando, Philippe Weinzaepfel, Grégory Rogez, Fabien Baradel
Affiliation: NAVER LABS Europe
Code: https://github.com/naver/anny
Area: Human Understanding
Keywords: parametric body model, interpretable shape space, anthropometry, synthetic data, human mesh recovery

TL;DR

Anny converts MakeHuman's artist-designed deformation assets into a differentiable body model without learning its shape space from scans, calibrates sampling with WHO statistics, and generates 780k Anny-One images, retaining millimeter-level adult scan fitting while improving child human mesh recovery.

Background & Motivation

Human Mesh Recovery (HMR) usually predicts the pose and shape parameters of a body model rather than arbitrary vertices, then uses that model to generate a complete mesh. The body model therefore determines both which bodies can be represented and which populations are favored by training data and optimization priors. The SMPL family learns compact shape spaces from scans and integrates readily with vision networks, but scan acquisition is costly, privacy-sensitive, and may underrepresent children, older people, and uncommon morphologies.

Adding more shape components does not necessarily solve this coverage problem. The paper notes that CAESAR participants are mainly adults from industrialized countries; a distribution learned from these scans is not automatically a global population distribution. Dedicated infant models such as SMIL, or interpolation between adult and child models, extend coverage but remain tied to existing scan assets. Meanwhile, the MakeHuman community has accumulated deformation assets for editing age, height, weight, muscle, and proportions, without directly targeting differentiable learning or standard HMR evaluation.

Anny uses these assets as an alternative body prior instead of training a larger scan-based model. It also separates the ability to represent a body from how frequently that body appears under random sampling: the former depends on the deformation space, whereas the latter requires population statistics. Core Idea: replace a scan-learned latent space with a continuous artist-designed deformation space, calibrate sampling using anthropometric statistics, and connect the same body model to scan fitting, image synthesis, and HMR through cross-topology mappings.

Method

Overall Architecture

Anny itself takes shape and pose parameters and outputs a posed human mesh; it is not an image-to-mesh network. The paper first implements differentiable body deformation, then calibrates the shape distribution and establishes vertex mappings to SMPL-X and HumGen3D, enabling both evaluation on existing benchmarks and synthesis of clothed characters.

For downstream learning, bodies are sampled from the calibrated distribution, poses are sampled from motion data, and Anny-One images are rendered. The prediction head of HMR2.0 or Multi-HMR is then adapted to output Anny parameters. This allows body representation, training imagery, and the regression network to be tested separately instead of attributing all system-level gains to a new network.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["MakeHuman assets<br/>Shape and pose parameters"] --> Shape["Differentiable Body Deformation"]
    Shape --> Prior["Statistical Shape Prior"]
    Prior --> Mapping["Cross-Topology Mapping"]
    Mapping --> Data["Synthetic Data and Mesh Recovery"]
    Data --> Output["Human meshes in images"]

Key Designs

1. Differentiable Body Deformation: continuous bodies from discrete prototypes

The default mesh contains 13,718 vertices, 13,710 quadrilateral faces, and 163 bones, including components such as the eyes and tongue. Sparse skinning weights softly attach vertices to bones. MakeHuman supplies 1,136 prototypical blendshape variations describing combinations of phenotype attributes. Anny applies piecewise multilinear interpolation between these prototypes, turning scalar controls into continuous shape changes rather than splitting people into separate categorical models.

Shape parameters alter the rest geometry and its associated shape-adjusted skeleton. Pose is specified by the root-bone pose and joint rotations relative to the rest configuration. Forward kinematics first determines bone poses, after which blend skinning produces the final mesh. PyTorch and NVIDIA Warp enable back-propagation, allowing geometric errors from scan fitting or HMR to reach the parameters. The central contribution here is making open deformation assets differentiable, not learning a new pose-corrective basis.

Phenotype names such as age and gender describe artist-defined shape controls, not measurements of a person's identity. Continuous interpretability makes parameter changes easier to understand and manipulate; it does not imply faithful encoding of real age, gender, or ethnicity. The paper explicitly warns that these semantics carry the designers' stereotypes.

2. Statistical Shape Prior: separate expressive range from plausible sampling

Uniform sampling over all controls can produce diverse bodies without ensuring a realistic population distribution. Anny first empirically defines a bijection between normalized age and morphological age in years, then models major phenotype parameters with Beta distributions conditioned on age and gender. Beta distributions suit bounded controls. Calibration adjusts these distributions rather than relearning all vertex assets.

Distribution parameters are jointly adjusted so that the mean and standard deviation of generated height and body mass index (BMI) match WHO growth standards. Figure 4 shows height-for-age and BMI-for-age calibration; Figure 5 checks the resulting weight-for-height relationship. This preserves the ability to represent extreme heights while allowing synthetic data to follow more realistic population statistics. Matching several summary statistics, however, does not establish the accuracy of the full population joint distribution.

3. Cross-Topology Mapping: bring a new body model into existing tools

Anny and SMPL-X have different vertex counts and connectivity, preventing direct vertex-wise evaluation; HumGen3D introduces another character representation for rendering. The paper learns sparse linear regressors between topologies. The following is equivalent to the vertex mapping in the original Section 3:

\[ \hat{\mathbf v}_i=\sum_j R_{ij}\mathbf v_j,\qquad R\in\mathbb R^{M\times N}. \]

The source mesh has \(N\) vertices and the target has \(M\) vertices, with each target vertex expressed as a weighted combination of a few source vertices. Anny is first fitted to meshes with the target topology. Target vertices are projected onto the source surface, and barycentric coordinates initialize the mapping. Mapping coefficients and Anny parameters are then jointly refined to minimize mesh-to-mesh distance while enforcing left-right symmetry. This represents positions on a continuous surface more naturally than simply matching nearest vertices.

The mean round-trip cyclic consistency error is 1.3 mm between SMPL-X and Anny and 1.7 mm between HumGen3D and Anny. These are conversion errors, not scan-fitting errors or HMR accuracy. The resulting interoperability lets existing benchmarks evaluate the new model while allowing clothed rendered characters to carry corresponding Anny geometry annotations.

4. Synthetic Data and Mesh Recovery: turn shape coverage into visual supervision

Each individual has an Anny ground-truth annotation and a corresponding HumGen3D character. Clothing and accessories come from the HumGen3D library, while Infinigen Indoors generates indoor scenes. Body poses come from AMASS, hand poses are independently sampled from GRAB, and shapes follow the calibrated distribution. A bounding volume hierarchy supports face-intersection checks across body parts, rejecting invalid bodies; scene placement also avoids intersections with surrounding objects. This is a data-generation filter, not a guarantee that every predicted mesh is intersection-free.

Scenes contain approximately 5 people on average, with up to 40 rendered camera views. The field of view is sampled uniformly from 30° to 130°, and selected images include egocentric, upper-body, and hand-focused views. Blender Cycles renders at 1280×1280 pixels, producing 780k images. Shape, viewpoint, and local-view diversity jointly determine the visual training coverage; changing the body parameterization alone does not automatically provide these data benefits.

HMR2.0 processes person crops using ViT-H with 16×16 patches and a single learned query that predicts parameters through cross-attention. Multi-HMR processes full images using a DINOv2-pretrained ViT with 14×14 patches and queries corresponding to detected people, predicting body parameters and three-dimensional placement. Only necessary adaptations are made to regress Anny parameters. Ablations use ViT-B with 448×448 inputs; comparisons with existing methods use ViT-L with 672×672 inputs.

A Worked Example

Consider a body halfway between the child and young prototypes. In the paper's Figure 3, age=0.5 gives equal contributions to child at age=1/3 and young at age=2/3. This is a deformation coordinate, not an age of half a year. Other controls determine height, weight, and proportions, and bone rotations place the resulting body into a sampled pose.

During synthesis, the same body is mapped to a HumGen3D character for clothing while retaining its Anny annotation. After self-intersection and scene-collision filtering, it is rendered alongside other people. Multi-HMR receives the complete scene image, with each person's Anny mesh and location as supervision. Evaluation in SMPL-X topology applies a mapping afterward, so rendering and evaluation meshes need not share connectivity.

Loss & Training

The body's foundational deformation assets are not trained on human scans. Fitting model parameters to a particular scan is distinct from learning the shape space from scans. The statistical prior is calibrated against WHO means and standard deviations, mappings are optimized using mesh-to-mesh distance, and HMR retains the existing networks' training frameworks. The cached main paper does not specify complete loss weights, optimizers, or training durations, so no unsupported training formula is added here.

Table 1 isolates the body-model effect by training both parameterizations on BEDLAM. HMR2.0 freezes its backbone and uses no data augmentation, while Anny shape modeling is restricted to 6 major controls: age, gender, height, weight, muscle, and proportions. The large-scale comparison in Table 3 instead mixes Anny-One, BEDLAM, MS-COCO, and MPII, so it is not a strictly data-matched body-model ablation.

Key Experimental Results

Main Results

Original Table 1 replaces the body model under the same BEDLAM training setup. All errors are in mm, lower is better. MPJPE is Mean Per Joint Position Error; PVE is Per Vertex Error; PA denotes Procrustes alignment before evaluation. The joint metrics belong to 3DPW and the vertex metrics to EHF, so their columns must not be interchanged.

Network	Body model	3DPW MPJPE	3DPW PA-MPJPE	EHF PVE	EHF PA-PVE
HMR2.0	SMPL-X	86.0	52.0	76.4	66.9
HMR2.0	Anny	86.5	49.4	65.5	49.7
Multi-HMR	SMPL-X	87.1	56.3	66.2	52.9
Multi-HMR	Anny	87.0	54.3	68.6	52.6

Not every metric improves: HMR2.0's 3DPW MPJPE changes from 86.0 to 86.5, and Multi-HMR's EHF PVE changes from 66.2 to 68.6. The supported conclusion is comparable performance without a scan-learned shape space, not universal superiority over SMPL-X.

The separate geometric validation in original Section 5 uses 400 adult scans from 200 people in 3DBodyTex. Excluding heads and hands, Anny achieves a mean scan-point-to-mesh error of 2.5 mm, versus 1.9 mm for SMPL-X with 300 shape components and a cited 1.8 mm for ATLAS. On three commercial child scans, mean errors are 2.0 mm for Anny and 3.2 mm for SMPL-X. These provide small-sample support, not a comprehensive child-scan benchmark.

Ablation Study

Original Table 2 analyzes Multi-HMR body models and training data on the authors' redefined AGORA validation set. All errors are in mm, lower is better. Images containing 3DPEOPLE scans form a 2k-image validation split, giving training and validation equal proportions of children. This is not the official validation split, which contains no children. Fine-tuning means training on AGORA; no pretraining with fine-tuning marked Yes means direct AGORA training.

Pretraining data	AGORA fine-tuning	Body model	All PVE	All PA-PVE	Kids PVE	Kids PA-PVE
BEDLAM	No	SMPL-X	140.8	71.3	186.0	64.2
BEDLAM	No	Anny	136.7	71.3	175.6	63.6
Anny-One	No	Anny	118.5	63.5	113.9	56.4
None	Yes	SMPL-X	89.7	60.8	99.5	50.6
None	Yes	SMPL-X+A	87.9	60.4	80.8	50.5
None	Yes	Anny	85.7	57.8	79.0	49.3
BEDLAM	Yes	SMPL-X	78.2	50.3	96.5	45.6
BEDLAM	Yes	SMPL-X+A	76.6	50.0	77.6	43.6
Anny-One	Yes	Anny	72.8	48.2	69.3	41.5

Key Findings

• Without AGORA fine-tuning and with Anny fixed, replacing BEDLAM with Anny-One reduces Kids PVE from 175.6 to 113.9 mm, a reduction of 61.7 mm. This changes both dataset content and scale, so the gain cannot be attributed solely to WHO calibration.
• With direct AGORA training, Anny achieves 79.0 mm Kids PVE versus 80.8 mm for SMPL-X+A, a smaller gap than against SMPL-X at 99.5 mm. This supports the importance of child-shape coverage and shows why the adult-child interpolation baseline matters.
• In the large-scale mixed-data comparison of original Table 3, CMU-Toddler MPJPE is 153.6 mm for Multi-HMR and 102.1 mm for Multi-HMR+Anny; Pair-PA-MPJPE is 638.9 and 263.8 mm, respectively. The latter jointly aligns each pair of interacting people to assess relative layout, rather than aligning each person independently.

Highlights & Insights

• Separating shape representation from population sampling avoids treating a scan collection's empirical distribution as the limit of model expressiveness. Rare shapes remain representable while an independent prior controls how frequently common shapes are sampled.
• Sparse mappings connect the new representation to existing evaluation and rendering assets, reducing the engineering cost of changing body models. Those mappings introduce errors of their own, so downstream results cannot be interpreted independently of conversion.
• Testing a data-matched body-model replacement before analyzing pretraining data helps separate contributions. The strongest child results come from the combination of representation and data, not the parameterization alone.

Limitations & Future Work

• The authors explicitly acknowledge artist priors and stereotypes in phenotype semantics. A scan-free design removes the need to acquire scans for shape learning, but does not eliminate population bias.
• Adult scan fitting remains weaker than the listed scan-based models, particularly for fine skin folds on high-BMI bodies. Anny does not model clothing, hair, or their surface deformations; lower scan error and a more accurate underlying body shape are not always the same objective.
• Child-scan evidence comprises only three examples, CMU-Toddler has limited child diversity, and AGORA is synthetic and re-split. Independent, diverse real-child evaluation is needed before claiming coverage of every age and morphology.
• The main paper does not provide separate numerical ablations for WHO calibration, self-intersection filtering, or individual sampling factors. A useful next step is to fix the data budget and isolate population-prior and scene-diversity effects.

Related Work & Insights

• vs SMPL-X / SMPL-X+A: the former learns shape from scans and the latter adds age-related interpolation; Anny uses artist prototypes and continuous semantic controls. Anny is more open and easier to control, but does not consistently improve adult geometric detail.
• vs MakeHuman: Anny inherits its assets and control semantics. Its contributions are differentiability, statistical calibration, cross-topology interoperability, and downstream validation, not a newly learned collection of prototype assets.
• vs BEDLAM: both supply synthetic HMR supervision. Anny-One combines the Anny body distribution with character rendering, indoor scenes, and wide field-of-view variation, suggesting joint design of body representation and training distribution rather than only scaling visual backbones.

Rating

• Novelty: 4/5, a clear integration of open assets, statistical priors, and HMR rather than a new neural architecture.
• Experimental Thoroughness: 4/5, covers scan fitting, body-model replacement, and multiple recovery benchmarks; real-child evidence and factor ablations remain limited.
• Writing Quality: 4/5, contributions are clearly separated, but detailed optimization settings are still needed for reproduction.
• Value: 5/5, an Apache 2.0 body model lowers the barrier to representation replacement and controllable data generation.