OmniFit: Multi-modal 3D Body Fitting via Scale-agnostic Dense Landmark Prediction

Conference: ECCV2026
Paper: ECCV Paper
Area: Human Understanding
Keywords: 3D body fitting, SMPL-X, dense landmarks, multimodal, scale prediction

TL;DR

OmniFit directly predicts dense, semantically fixed body landmarks from clothed-human point clouds before optimizing SMPL-X, with an optional RGB adapter and scale prediction for appearance cues and scale ambiguity, reducing 4D-DRESS point-cloud V2V from ETCH's 2.408 cm to 0.589 cm.

Background & Motivation

A human scan records the outer surface formed by clothing, hair, and the body, whereas animation and motion transfer require an animatable body underneath the clothes. Moving a body template toward the nearest scan points therefore does not necessarily recover the correct body: loose garments can lie far from the torso, while fingers and the face require finer geometric constraints. Traditional multi-view fitting commonly renders the scan, detects 2D keypoints, triangulates 3D keypoints, and finally optimizes a parametric body. This process exploits texture, but detection errors propagate downstream; untextured assets and inputs visible from only one side are also poorly suited to the complete rendering pipeline. Learning-based methods supply fitting evidence from point clouds, often using joints, correspondences, or sparse markers as intermediate representations to avoid the difficulty of direct body-parameter regression. ETCH already aggregates dense correspondences into compact markers, but this aggregation depends on observed surface points, so missing regions can also cause their markers to disappear.

OmniFit targets inputs beyond standardized scans: real RGBD observations, synthetic clothed humans, and generated 3D assets may all enter the same fitting system. Their problems differ: the point cloud may lack a back surface, RGB may or may not be available, and generated assets may have no reliable metric scale. Normalizing every asset to the same bounding box standardizes its numerical range rather than its body size, because seated and standing poses occupy different extents. Meanwhile, facial and hand cues in images should not become a mandatory requirement for processing untextured point clouds. The paper preserves an independently usable 3D backbone and introduces explicit interfaces for scale and appearance information instead of requiring identical capture conditions for every input.

Its intermediate representation consists of dense landmarks predefined on the SMPL-X template, with each output always representing the same body location. The input point cloud supplies evidence, but the identity and number of output landmarks do not depend on the observed surface, so missing observations do not directly remove prediction targets. Hands and the head can receive more target locations, RGB adds conditioning only when available, and scale restoration precedes the landmark backbone. Core Idea: connect clothed-surface understanding to parametric body fitting through dense landmarks with fixed semantics, while separating optional appearance conditioning and scale restoration from core geometric prediction.

Method

Overall Architecture

The primary input is a 3D point cloud sampled from a mesh, taken from 3D Gaussian Splatting point positions, or captured by an RGBD sensor; a single front-view RGB image is optional. The output is an SMPL-X body, not a reconstruction of the input clothing surface or merely a skeletal pose estimate. SMPL-X represents the body through shape, pose, expression, and global translation; the paper specifies a mesh with 10475 vertices and a kinematic structure containing 55 joints. Shape and expression each use 10 coefficients, and pose includes the body, eyes, jaw, and fingers, so fitting constraints must cover both large limbs and detailed regions. When scale is unknown, Scale Restoration converts the normalized point cloud to canonical human size before Dense Landmark Decoding. Optional Image Adaptation operates inside that decoder, after which Landmark-Constrained Fitting converts predicted coordinates into valid SMPL-X parameters. In the diagram, point-cloud and image conditions are fused in parallel within decoder layers; training supervision is not an inference-time input.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    Input["Human point cloud"] -->|Unknown scale| Scale["Scale Restoration"]
    Input -->|Known canonical scale| Point["Point-cloud encoding"]
    Scale --> Point
    subgraph Decoder["Dense Landmark Decoding"]
        direction TB
        Query["600 learnable queries"] --> Fusion["Condition fusion and self-attention"]
        Adapter["Optional Image Adaptation"] -->|Parallel per-layer condition| Fusion
        Fusion --> Landmarks["600 3D landmarks"]
    end
    Point --> Fusion
    Image["Optional front-view RGB"] --> Adapter
    Landmarks --> Fit["Landmark-Constrained Fitting"]
    Fit --> Output["SMPL-X body"]

Key Designs

1. Scale Restoration: resolve the mismatch between input size and the body prior first

Generative models often compress human assets into a unit bounding box, but seated and standing bodies inside the same box do not represent the same physical height. OmniFit feeds the normalized point cloud to a separate scale predictor and prepends a learnable scale token to its point-cloud patch-token sequence. After the Transformer, an MLP reads only the scale-token output to regress a scale factor, rather than independently correcting each coordinate. Section 3.4 applies the predicted scale as follows:

\[ X=\widetilde{X}\cdot S. \]

Here, \(\widetilde{X}\) is the normalized input, \(S\) is the predicted uniform scale, and \(X\) is the point cloud passed to the landmark predictor. Training supervision is constructed by normalizing original humans to \([-0.9,0.9]\), with the target factor restoring their pre-normalization size. The module addresses global scale ambiguity; it does not repair local stretching, distorted body proportions, or arbitrary geometric corruption. From an identifiability perspective, normalized geometry alone cannot uniquely determine true metric height, so this should be understood as learning plausible human scale from the training distribution rather than recovering unobservable physical ground truth.

2. Dense Landmark Decoding: fixed output locations with variable observational evidence

The model predicts 600 predefined SMPL-X template vertices: 120 on the head, 300 on the body, and 180 across both hands. These landmarks are neither samples of the scanned surface nor centers produced by clustering the current input; they are fixed semantic targets on the underlying body. A Point-BERT backbone converts the input point cloud into patch embeddings and then extracts local point-cloud features through Transformer layers. A Perceiver-style decoder uses 600 learnable embeddings as queries, with each layer reading point-cloud evidence through cross-attention and exchanging landmark information through self-attention. Cross-attention identifies observations supporting each body location, while self-attention coordinates predictions for the head, torso, and limbs. A final MLP outputs a 3D coordinate for each query, giving the fitting stage a fixed-size target set with consistent identities.

Crucially, queries still exist when observations of the back or a hand are missing, allowing the model to infer their locations from the remaining geometry and body priors. This does not mean the occluded region has been measured; it means the output interface no longer fails simply because part of the surface is absent. Figure 9 groups attention visualizations by head, body, and hands and shows queries attending to corresponding input regions, supporting the interpretation of implicitly learned part correspondence. However, these attention maps are qualitative evidence and cannot establish precise one-to-one correspondence for every point. Compared with querying an implicit correspondence field point by point, directly supervising a finite set of fixed targets also clarifies the training objective and permits deliberate allocation of constraints to detailed regions.

3. Optional Image Adaptation: add appearance evidence inside geometric decoder layers

When an asset has texture, or an RGBD capture supplies a color image, the model can additionally use one front-view RGB image. DINOv2 extracts image features, and a lightweight image cross-attention branch runs in parallel with the original point-cloud cross-attention in every decoder layer. Landmark queries read both conditions before residual fusion and self-attention, so the image supplements rather than replaces point clouds, especially where surface geometry is ambiguous. Finger poses and local facial structures are emphasized beneficiaries; clothing texture can also provide cues unavailable from sparse point positions alone. Disabling the adapter retains the original point-cloud model, so RGB is not required for every input and the system is not a purely single-image body-recovery method.

Training proceeds from backbone to adapter: the point encoder and landmark decoder are first trained jointly, after which the existing predictor is frozen and only the added adapter branch is updated. The adapter uses the same landmark-coordinate supervision, without replacing the target with image generation or pixel reconstruction. For 4D-DRESS, its training condition comes from front-view orthographic renderings of textured meshes, making the evaluated image configuration explicit. The paper does not provide a complete evaluation for arbitrary camera poses, multi-view sets, or severe image–point-cloud mismatch; plug-and-play should not be interpreted as support for arbitrary visual inputs.

4. Landmark-Constrained Fitting: project coordinate predictions back into an animatable body space

The landmark network outputs coordinates without directly guaranteeing that one valid body model can generate all of them. At inference time, SMPL-X shape, pose, expression, and translation are optimized so that corresponding model landmarks approach the predicted landmarks. Section 3.5 describes the objective as the sum of squared Euclidean distances between corresponding landmarks, using fixed vertex identities without another nearest-neighbor matching step. Section 4.3 specifies three optimization stages: translation alone for 20 steps at learning rate \(5\times10^{-1}\); the first 2 shape coefficients and pose jointly for 30 steps at \(5\times10^{-1}\); and all parameters for 20 steps at \(2\times10^{-1}\). Aligning position first, then major shape and motion, and finally all degrees of freedom can prevent local pose changes from absorbing initial translation errors; this is a mechanistic interpretation of the schedule. The total of 70 optimization steps also means that learning-based fitting does not produce the final body in a single forward pass: iterative cost remains.

A Worked Example

Consider a seated human normalized to \([-0.9,0.9]\), with only front-visible points and an accompanying front-view RGB image; this illustrates the workflow rather than reporting an additional experiment. The scale predictor uses the seated geometry to output \(S\) and uniformly resize the point cloud to the expected human scale, rather than interpreting bounding-box height as standing height. The point encoder supplies local geometry, and 600 queries seek evidence for their respective body locations; landmarks on the unseen back are still predicted. If enabled, the image adapter contributes hand and face information within decoder layers, while the output remains the same set of 600 coordinates. The subsequent 20-, 30-, and 20-step optimization stages produce an SMPL-X body usable for animation, rather than treating the generated asset's clothing as skin. Without an image, the image branch is skipped; with an already canonical-scale input, scale estimation is unnecessary.

Loss & Training

Landmark prediction uses mean squared error between predicted and ground-truth landmark coordinates; the independently trained scale predictor uses mean squared error between predicted and ground-truth scale factors. Several extracted equations contain damaged macros and operators, so the objectives are explained from the prose in Section 3.5 rather than presenting repaired expressions as exact author equations. The point encoder, landmark decoder, and scale predictor use 16, 24, and 12 Transformer blocks, respectively. Training uses AdamW at learning rate \(5\times10^{-5}\) for 100 epochs on 8 NVIDIA GPUs with 48GB memory each. Per-GPU batch sizes are 20 for the landmark predictor, 24 for the adapter, and 64 for the scale predictor. Training point counts are sampled from \([5000,20000]\) with random rotations, and 50% of full point clouds are replaced by partial ones. Partial point clouds are created by removing vertices and faces invisible from the front view and sampling the remaining surface, giving the missing-data pattern an explicit single-view origin. Standard comparisons follow ETCH's CAPE and 4D-DRESS splits; the stronger generalizable model additionally uses two synthetic human datasets. Augmented BEDLAM2 contributes 98,659 frames, while retargeting Motion-X motions to SynBody templates yields 158,410 frames. Synthetic data expand pose and geometry coverage while supplying accurate underlying-body supervision, serving fitting more directly than appearance diversity alone. The generalizable image adapter additionally uses point clouds and front-view renderings from 2K2K and X-Humans and should not be confused with the adapter trained only on 4D-DRESS.

Key Experimental Results

Main Results

Source: Table 1, page 9; point-cloud-only input with 5,000 points throughout, author-implemented SMPL-X versions of the baselines, and matching training and testing settings. V2V measures mean Euclidean distance between corresponding mesh vertices; MPJPE measures mean SMPL-X joint-position error. All entries below are whole-body errors in cm, lower is better.

Method	CAPE V2V	CAPE MPJPE	4D-DRESS V2V	4D-DRESS MPJPE
IPNet	5.529	5.611	7.495	7.380
PTF	2.341	2.641	3.297	3.567
NICP	1.736	2.074	4.085	4.862
ArtEq	2.202	2.405	3.072	3.378
ETCH	1.567	2.002	2.408	3.459
OmniFit	0.672	0.659	0.589	0.662

Relative to ETCH, V2V/MPJPE reductions are 57.1%/67.1% on CAPE and 75.5%/80.9% on 4D-DRESS, consistent with the whole-body entries. An error of 0.589 cm equals 5.89 mm; millimeter-level accuracy refers to benchmark mean errors, not a guarantee for every real input. Table 2, page 11, shows that adding the image adapter on 4D-DRESS reduces V2V/MPJPE from 0.589/0.662 to 0.539/0.622 cm. The same table reports 3.047/4.029 for EasyMocap and 2.093/2.630 cm for DiffProxy, but these methods use multi-view RGB and camera information, whereas OmniFit uses geometric point clouds with optional front-view imagery. The authors explicitly state that this is not a strictly modality-matched comparison; it supports 3D-native fitting when 3D assets are available, not the claim that a single image contains more information than multiple views.

Ablation Study

Source: Table 4, page 12; the landmark predictor is trained on 4D-DRESS, and inputs are normalized to \([-0.9,0.9]\) to evaluate scale handling. Errors are in cm, lower is better.

Scale configuration	V2V	MPJPE	Interpretation
Ground-truth scale	0.589	0.662	Reference with correct known scale
Without scale prediction	1.120	1.172	No predictor-based scale restoration
With scale prediction	0.651	0.709	Predicted scale applied before fitting

Scale prediction reduces V2V from 1.120 to 0.651 cm, a reported reduction of 41.9%; the reported MPJPE reduction is 39.5%. It does not reach the ground-truth-scale reference, showing that scale estimation is effective but does not eliminate all scale-related error.

Source: Table 5, page 12; the total is fixed at 600 landmarks, with different regional allocations evaluated on 4D-DRESS. Whole-body V2V/MPJPE are in cm.

Configuration	Hand landmarks	Head landmarks	Body landmarks	V2V	MPJPE
A	300	120	180	0.655	0.701
B	240	120	240	0.621	0.690
C, adopted	180	120	300	0.589	0.662
D	120	120	360	0.637	0.724

Allocating more targets to hands does not necessarily yield the best whole-body fit; configuration C empirically better balances body-constraint coverage and hand detail. The regional improvement row in Table 1 does not always use the best baseline in that column: for CAPE hand V2V, the best baseline is NICP at 2.741, whereas the stated 80.6% corresponds to ETCH at 3.449. Its per-region second-best interpretation is therefore not repeated here.

Key Findings

• Input density matters: Table 7, page 13, increases the point count from 5,000 to 15,000 and reduces V2V from 0.589 to 0.452 cm; this denser-input result should not be mixed with the uniform 5,000-point comparison in Table 1.
• Fixed queries are not free: Table 6, page 13, increases landmarks from 100 to 600, raising computation from 23.82 to 93.38 GFLOPs and reducing training throughput from 0.88 to 0.46 iterations per second; this throughput is not inference frame rate.
• Partial inputs still lose information: Table 9, page 13, increases CAPE V2V from 0.672 for full inputs to 0.887 cm, while 4D-DRESS changes from 0.589 to 0.595 cm.
• Adapter generalization depends on training coverage: Table 3, page 11, reports THuman2.1 V2V of 1.767 for Adapter and 0.835 cm for Adapter*, where the asterisk denotes extended training data.

Highlights & Insights

• Fixed semantic queries separate the existence of an observation from the body location that must be predicted. This is particularly valuable with missing regions, but inferred completion must still be distinguished from measurement.
• Dense landmarks provide an inspectable intermediate representation. They support direct coordinate supervision and explicit optimizer correspondences, making errors easier to localize than when the entire task is compressed into body-parameter regression.
• The optional RGB branch preserves the standalone utility of the geometric model. Supplementing a frozen backbone through a modality-specific adapter is a transferable design for other geometric fitting tasks.

Limitations & Future Work

• On page 15, the authors explicitly acknowledge the inference-time cost of iterative SMPL-X optimization; the current method is not an optimization-free real-time parameter regressor.
• Their other stated limitation is support for only one front-view image, with multi-view adaptation left to future work.
• Real-capture evidence is small: Table 10, page 14, tests only 20 BEHAVE Kinect captures, with V2V of 2.550 and 2.224 cm for the generalizable point-cloud model and adapter.
• As a reader interpretation, scale prediction depends on body-size priors, and partial-input training depends on single-view visibility simulation; unusual body types, special occlusions, and nonuniform distortions require separate evaluation rather than automatic extrapolation from these results.
• A possible extension is uncertainty-aware landmark prediction, using confidence to weight fitting constraints and reduce the impact of incorrect invisible-region estimates; this is a research suggestion, not an implemented component.

Related Work & Insights

• ETCH, source reference [31], page 17: aggregates correspondences into markers; OmniFit directly predicts landmarks through fixed queries, so target identity does not depend on aggregating the current observations.
• Perceiver, source reference [26], page 17: supplies the architectural foundation for reading input features with learnable queries; OmniFit turns this into a coordinate-output interface with SMPL-X vertex semantics.
• SMPL-X, source reference [47], page 18: provides a unified parametric space for the body, hands, and face; OmniFit improves the prediction of constraints from clothed observations into that space rather than changing the body model itself.
• EasyMocap and DiffProxy, source references [1]/[61], pages 16/19: represent multi-view fitting pipelines; the lesson here is to exploit available 3D geometry directly rather than first converting all evidence into 2D observations.

Rating

• Novelty: 4/5. Fixed dense landmarks, optional image adaptation, and scale restoration form a clear task-specific solution, while the underlying attention components build on existing architectures.
• Experimental Thoroughness: 4/5. Standard comparisons, scale and configuration ablations, cross-dataset evaluation, and partial-input tests are included, but real-capture scale and modality fairness remain limited.
• Writing Quality: 4/5. The method is clearly organized and comparison boundaries are acknowledged; extracted equations are damaged, and the regional improvement description requires careful checking.
• Value: 4/5. Useful for turning scanned or generated humans into animatable models, with deployment value still dependent on optimization latency and the actual capture distribution.