Revisiting Avatar-As-Image: High-Fidelity Registration is All You Need

Conference: ECCV2026
Paper: ECCV Paper
Project: AvaImg
Authors: Margaret Kostyrko, Yuxuan Xue, Garvita Tiwari, Gerard Pons-Moll
Area: Human Understanding
Keywords: human registration, signed winding numbers, shape under clothing, coarse-to-fine displacement, UV representation

Identity check: the manifest and cached first page both correspond to official ID 3431, with matching title and subtitle. This note follows the conference PDF; its promise to release code, data, and Singularity containers is not treated as an already available code repository.

TL;DR

AvaImg first places a plausible naked body inside clothing, progressively fits the garment surface, and converts the result into aligned UV images, achieving 2.62 mm surface Chamfer distance and 34.48 dB rendering PSNR in the main comparison, while a frozen FLUX VAE roundtrip adds only 0.76 mm geometry error.

Background & Motivation

Representing a 3D human as 2D UV images is an established idea: SMPL and SMPL-X provide shared topology, allowing a UV pixel to refer to the same anatomical location across subjects. Texture maps encode color, while displacement maps capture clothing geometry beyond the body. These images could feed existing image networks, but inaccurate scan-to-template registration misaligns folds, silhouettes, and colors before a downstream model ever sees them.

Existing tools also conflate two objectives: estimating the hidden naked body and recovering the observed garment surface. Under unsigned distance minimization, approaching the clothing from inside and penetrating back toward it from outside can look equally good. Dataset-provided fits are not necessarily reliable ground truth, particularly around hands, feet, and open scan regions. Learning from these registrations perpetuates the same errors.

The paper therefore improves the prerequisite representation rather than introducing another generator. Its single-scan optimization pipeline produces accurate, textured registrations with consistent topology. Core Idea: separate body containment from clothing-surface reconstruction, first estimating a more physically plausible body and then producing aligned UV texture and displacement maps for image-based models.

Method

Overall Architecture

The input is a textured clothed-human scan with vertices, triangles, and UV coordinates. The output is a SMPL(-X)+D registration together with standardized UV texture and displacement maps. D denotes per-vertex displacement, not a new generative network. Robust Pose Initialization, Winding-Constrained Fitting, Progressive Displacement Refinement, and Unified UV Mapping form the four stages in Figure 2 and Sections 3.1-3.4.

The important transition is what each stage should approach: body fitting discourages the naked body from escaping the garment, whereas surface registration must let displacements leave the body and reach the clothing shell. Throughout optimization, the complete SMPL(-X) manifold and regional priors help prevent holes, noise, and scan contacts from directly determining the output topology.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Scan["Clothed-human scan"] --> Init["Robust Pose<br/>Initialization"]
    Init --> Fit["Winding-Constrained<br/>Fitting"]
    Fit --> Refine["Progressive Displacement<br/>Refinement"]
    Refine --> UV["Unified UV Mapping"]
    UV --> Output["Texture and displacement maps<br/>Editable avatar"]

Key Designs

1. Robust Pose Initialization: establish joints before fitting surfaces

A mean-pose initialization can trap raised arms or crossed legs in incorrect local minima. AvaImg first normalizes the scan to the gendered SMPL(-X) model using height-based scaling and centroid translation. PyTorch3D then renders 72 views over 3 elevation levels with full azimuth coverage, giving each joint multiple opportunities to be visible.

OpenPose detects 137 keypoints per view, and multiview bundle adjustment lifts these detections into 3D joint targets. Instead of averaging every observation, the procedure combines an L1 residual, squared confidence weighting, and a hard confidence threshold of 0.3. Unreliable detections receive zero weight, reducing the influence of occlusion and incorrect correspondences.

Joint fitting also releases parameters progressively. It first optimizes global orientation, the head, shoulders, and right foot; then all body joints; and finally the complete model, including hands and facial expression. Large-scale alignment is established before high-degree-of-freedom extremities, providing a reliable starting point for body fitting.

2. Winding-Constrained Fitting: distinguish inside and outside distances

With pose initialized, the pipeline estimates shape beneath clothing. Generalized winding numbers determine containment volumetrically instead of relying only on the nearest surface normal or a single ray, making the decision more robust to noisy and non-watertight scans. The paper assigns +1 to inside and -1 to outside, incorporates this sign into mesh-to-scan distance, and uses Geman-McClure robustness with asymmetric pReLU weighting: internal vertices receive a mild penalty, while escaped vertices are penalized more strongly.

Equation (1) is corrupted in the text extraction, so this note does not invent its complete algebraic form or pReLU slopes. The supported mechanism is an inside/outside-aware distance combined with joint, shape, and pose priors whose weights are annealed. This is a soft optimization constraint, not a guarantee of zero collision; the reported results retain nonzero penetration.

Naively evaluating winding numbers scales with the product of query-point and scan-face counts. AvaImg first decimates the scan to approximately 10% of its original faces, subject to a 40k-face minimum. It then precomputes winding numbers on a 5 mm voxel grid around the bounding box and thresholds them into a binary sign field. Body fitting subsequently uses nearest-neighbor queries rather than repeated high-resolution triangle calculations.

Finally, winding band reduction retains only boundary voxels whose 7 nearest neighbors include a sign change, discarding roughly 95% of the grid. The cascade reduces triangle workload, repeated computation, and field storage respectively. It compresses the containment representation; it does not permanently replace the detailed scan surface needed for later refinement with the decimated mesh.

3. Progressive Displacement Refinement: fit the silhouette before releasing fine folds

Once the body is plausible, the goal changes to recovering clothing, hair, and shoes. The pipeline removes the containment constraint and switches to unsigned data terms. The first pass optimizes free vertices initialized from the body mesh, with posed-to-canonical displacement consistency reducing dependence on articulation. Edge coupling penalizes changes relative to the undisplaced body, while cotangent Laplacian smoothing suppresses noise.

Regularization is spatially selective rather than uniform. Bone-based skinning weights determine regions for edge coupling, such as restricting deformation around hands, and designed vertex maps control smoothing. Strong early regularization captures the global silhouette; annealing then permits finer deformation. This is particularly important when missing hand geometry would make unrestricted nearest-surface fitting unreliable.

The second pass applies Loop subdivision, approximately quadrupling the face count. At this resolution, the data term uses only scan-to-mesh distance, with an increasing multiplier to tighten adherence to the outer garment surface. Regional weights are adjusted for important fine structures. The main text does not supply all numerical weights or regional maps, so these descriptions alone do not constitute a complete reproduction recipe.

High-resolution edge coupling changes from a squared to a fourth-power penalty. Using the notation given in Section 4.5:

\[ E_{\mathrm{coupling}}=(w_c L_c)^2 \quad\longrightarrow\quad E_{\mathrm{coupling}}=(w_c L_c)^4. \]

For small weighted residuals, the fourth-power penalty has a flatter basin that permits local detail; for large residuals, its steeper sides discourage substantial topological distortion. The intention is to preserve folds, garment edges, and heels instead of smoothing them away. This change is distinct from increasing resolution, and the ablation measures their separate benefits.

4. Unified UV Mapping: turn geometric correspondence into reusable image coordinates

The final registrations share the standard SMPL(-X) UV topology. An f-map records the high-resolution template triangle covering each UV pixel, and a b-map records its barycentric coordinates. These lookup maps depend only on topology and target resolution, so they can be reused across subjects. Switching from \(512^2\) to \(4096^2\), or changing the model variant, requires a one-time recomputation.

For each covered pixel, the lookup maps locate a 3D point on the registered mesh. The pipeline finds its nearest point on the scan, expresses that point in the corresponding scan triangle, and bilinearly samples color in the scan's native UV coordinates. Morphological inpainting fills uncovered pixels. For vertex-colored scans, colors are interpolated directly from nearby scan vertices. Applying the same correspondence procedure to per-vertex displacements produces the UV displacement map.

The result is not simply a mesh flattened into arbitrary images: identical pixels now refer to anatomically corresponding regions across subjects. This enables texture transfer, shape editing, and re-animation, although nearest-surface transfer can still cause bleeding in self-contact regions. The frozen FLUX VAE experiment tests representation compatibility under compression; it does not train a new avatar diffusion model.

A Worked Example

Consider an illustrative raised-arm scan with clothing folds, not an additional reported test case. The 72 views establish 3D joints while detections below 0.3 confidence are excluded. The sign field then distinguishes a torso remaining inside the shirt from one penetrating outward, even when both are similarly close to the observed surface.

Low-resolution displacement captures the garment silhouette, and the subdivided second pass recovers folds. Unified UV coordinates transfer color and displacement into aligned maps. Passing these maps through a frozen VAE tests how much detail survives encoding and decoding, not whether the model can generate a new garment.

Loss & Training

This is a per-scan optimization pipeline rather than a newly trained registration network. Initialization uses joint fitting with shape and pose priors; body fitting adds signed distance; low-resolution surface fitting uses unsigned distance, displacement consistency, edge coupling, and Laplacian terms; high-resolution fitting switches to one-way scan-to-mesh distance, fourth-power coupling, and revised regional weights.

The main text does not provide a complete optimizer specification, learning rate, per-stage iteration counts, robust scales, or regularization-weight schedule. The full pipeline takes approximately 25 minutes per scan, including joint estimation, winding computation, body and surface fitting, and texture mapping. A speedup for one component should not be presented as an end-to-end speedup.

Key Experimental Results

Main Results

The six datasets contain 47 subjects from 4D-Dress, 26 from BuFF, 40 from CAPE, and 20 each from THuman2.1, 2K2K, and CustomHuman. The main method comparison uses the first three, totaling 113 subjects. Under-clothing shape evaluation uses BuFF minimal-clothing scans, with T-pose shapes aligned by Procrustes analysis. Surface Chamfer uses 100k samples per mesh.

Penetration rate is the fraction of body vertices outside the scan; penetration depth averages their distance to the scan surface; proximity averages body-to-scan distance. Low proximity alone is not evidence of a correct body because a naked body should have some offset from clothing. The following values are from Table 3, with all distances in mm. NR means not reported, not zero.

Method	Penetration (%)	Penetration depth	Proximity	Body-shape Chamfer	Surface Chamfer
IPNet	44.5	25.93	23.41	9.95	8.61
ETCH	35.0	13.53	13.01	7.76	NR
NICP	54.6	12.98	12.03	8.84	3.06
PTF	48.9	8.17	8.68	8.85	6.92
RMR	43.2	7.80	8.60	10.54	3.42
AvaImg	19.8	3.62	9.47	7.72	2.62

Table 3 reports texture-rendering PSNR of 34.48 dB, and Section 4.3 additionally reports FID of 5.19. The baselines do not support equivalent texture remapping, so this is not a controlled texture-quality ranking against them. The prose gives RMR penetration as 43.3%; both editions of this note use Table 3's 43.2%.

Ablation Study

The following results come from Figure 6 and Section 4.5. The first two rows change body-fitting distance, whereas the last three change surface refinement; they are not one continuous module-addition sequence.

Ablation group and configuration	Penetration (%)	Body-shape Chamfer (mm)	Surface Chamfer (mm)	Source
Body fitting: unsigned distance	40.7	11.95	NR	Figure 6
Body fitting: signed distance	19.8	7.72	NR	Figure 6
Surface: low resolution only	Unchanged	Unchanged	3.15	Section 4.5
Surface: two passes, squared coupling	Unchanged	Unchanged	2.75	Section 4.5
Surface: two passes, fourth-power coupling	Unchanged	Unchanged	2.62	Section 4.5

Signed fitting reduces penetration by 20.9 percentage points. Full refinement reduces surface error by 0.53 mm relative to low resolution only. In an efficiency example, decimating a roughly 370k-face scan to 40k faces reduces winding computation from approximately 2,400 s to 240 s. Band extraction adds about 70 s, while body fitting falls from approximately 1,700 s to 240 s; the related stages together fall from approximately 4,000 s to 540 s. Storage drops from roughly 70-100 MB to 5 MB. This is a profiled example, not a dataset-wide latency distribution.

The frozen FLUX VAE roundtrip results below are from Table 4. Spatial CD and rendering metrics use the original scan as reference; UV PSNR uses the original UV texture.

Configuration	CD to scan (mm)	Rendering PSNR (dB)	Rendering SSIM	Rendering LPIPS	UV texture PSNR (dB)
Before VAE	3.15	34.48	0.995	0.006	N/A
After VAE	3.91	30.04	0.988	0.009	38.6

Table 4 also reports UV texture SSIM of 0.964, displacement RMSE of 4.98 mm, and spatial V2V of 4.29 mm. Its CD baseline is 3.15 mm, not Table 3's 2.62 mm. The text does not adequately explain this setup difference, so the 0.76 mm increment is calculated only as 3.91 minus 3.15 within Table 4.

Key Findings

• Containment metrics distinguish methods better than proximity alone: RMR has lower proximity but substantially higher penetration, so its smaller distance does not establish a more accurate body.
• Table 2 compares dataset-provided registrations on five datasets, reporting 28.2% to 15.0% penetration; this is a different aggregation from Table 3's three-dataset method comparison.
• VAE compatibility is not losslessness: rendering PSNR still decreases from 34.48 to 30.04 dB.

Highlights & Insights

• Assigning body containment and garment-surface fitting to different stages is the most transferable modeling decision. One undifferentiated distance objective has no reason to distinguish naked-body geometry from clothing.
• Precomputing a sign field moves expensive geometric decisions out of the optimization loop. Boundary-band storage further shows that preserving containment information need not require a complete dense volume.
• The paper treats registration as training-data infrastructure rather than only a reconstruction metric. Accurate shared correspondence provides a reliable interface for texture editing and image-based generative priors.

Limitations & Future Work

• The authors acknowledge that SMPL(-X)+D topology limits loose or detached garments. Complex nearby surfaces can be associated incorrectly, self-contact can cause texture bleeding, and unposing bent joints can introduce indentations.
• Main-comparison penetration remains 19.8%, so the paper's absolute language about eliminating penetration should not be repeated literally. Soft constraints, incomplete scans, and template restrictions warrant further analysis.
• Approximately 25 minutes per scan suits offline reference-data creation, not real-time replacement of feed-forward methods. Training faster predictors from these registrations is plausible, but the paper does not validate that downstream gain.
• Successful frozen-VAE reconstruction supports representation compatibility, not a strict proof that UV maps belong to the natural-image distribution. Novel-subject generation still requires diffusion training and evaluation.
• Missing optimization details affect reproducibility. The unexplained CD baseline difference between Tables 3 and 4, and the single-example efficiency profile, also limit stronger cross-experiment conclusions.

Related Work & Insights

• vs RMR: RMR primarily uses unsigned registration and lacks UV texture output. AvaImg adds reliable containment decisions, SMPL-X support, and texture transfer, at the cost of per-scan optimization.
• vs ETCH, NICP, IPNet, and PTF: learned body fitting or surface registration emphasizes prediction efficiency, whereas AvaImg aims to improve the registration supervision these approaches depend on. ETCH does not output a garment surface, so its missing surface metric is not a failed reconstruction.
• vs ClothCap and BuFF: related pipelines use multiple frames or normal information to infer body-clothing relationships. AvaImg targets single scans with volumetric inside/outside reasoning.
• vs Tex2Shape, SMPLitex, and FLUX: UV geometry, texture modeling, and image priors are not new contributions here. The added value is higher-fidelity aligned representations and a demonstration that a frozen image VAE can process them without fine-tuning.

Rating

• Novelty: 4/5. Contributions lie in the combination of registration constraints, efficiency, and refinement, not in inventing Avatar-As-Image.
• Experimental Thoroughness: 4/5. Six datasets, complementary body and surface metrics, and key ablations are covered, but downstream generation remains untested.
• Writing Quality: 3/5. The stages are clear, while absolute claims and insufficient cross-table setup explanations require care.
• Value: 4/5. Accurate shared registration could improve training data and avatar editing; practical usability still depends on the complete implementation and release.