DiffProxy: Multi-View Human Mesh Recovery via Diffusion-Generated Dense Proxies

Conference: ECCV2026
Paper: ECCV Paper
Project: DiffProxy
Area: Human Understanding
Keywords: human mesh recovery, SMPL-X, dense correspondence, hand refinement, uncertainty weighting

Identity check: manifest ID 3382 matches the cached first-page title and research content. The authors are Renke Wang, Zhenyu Zhang, Ying Tai, Jun Li, and Jian Yang. No subtitle change was found; missing ligatures in the extracted word DiffProxy are extraction artifacts, not a different paper. The cache comes from the official entry's linked PDF; PDF filename number 1213 is not the conference entry ID.

TL;DR

DiffProxy generates body segmentation and UV correspondences with a diffusion model, then recovers a human mesh through hand refinement, stochastic uncertainty estimation, and SMPL-X reprojection fitting, reaching 35.1 mm MPVPE on RICH with synthetic-image training alone, without winning every metric on every dataset.

Background & Motivation

Human mesh recovery can directly regress SMPL-X parameters or first detect image evidence and then optimize a body model. Direct regression is fast but combines camera geometry, occlusion, and pose into one target, making misalignment difficult to diagnose. Fitting systems such as EasyMoCap triangulate multiview keypoints and fit a model, providing more explicit constraints. However, sparse joints leave much of the surface unconstrained, and one incorrect detection can pull an entire arm into the wrong configuration.

DensePose offers a different intermediate representation: associate every visible human pixel with a point on a canonical body surface. Historically, noisy manual correspondence annotations and limited generalization of CNN predictors weakened this approach on unusual poses and complex images. DiffProxy revisits it using accurately rendered synthetic supervision and the visual priors of Stable Diffusion 2.1. Its target is high-precision offline reconstruction, not real-time human tracking.

Core Idea: replace opaque image-to-body-parameter regression with inspectable dense correspondence generation, and convert disagreement between stochastic predictions into reliability weights for geometric fitting.

Method

Overall Architecture

The inputs are multiview RGB images of one person and calibrated cameras; the output is an SMPL-X mesh with body and hand poses. A dense proxy generator first predicts full-body proxies, locates hands in their segmentation, and jointly processes enlarged hand crops as additional views. Multiple stochastic outputs are aggregated before optimizing a shared body model so that its surface points reproject onto their corresponding pixels in every view.

A proxy consists of two three-channel images rather than directly predicted 3D vertices. Segmentation encodes body parts with fixed colors, while the UV image stores surface coordinates in its first two channels and one in its third channel. The part label and UV jointly locate a point on the canonical SMPL-X surface. The three-channel format matches the SD VAE, and generated proxies have resolution 256x256.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Multiview images and cameras"] --> Generator["Dense proxy generation"]
    Generator --> Hands["Hand refinement"]
    Hands --> Uncertainty["Sampling and reliability estimation"]
    Uncertainty --> Fit["Weighted geometric fitting"]
    Fit --> Output["SMPL-X body and hands"]

Key Designs

1. Dense proxy generation: combine exact synthetic supervision with pretrained visual priors

The training set combines 67,650 BEDLAM-derived samples and 37,837 SynBody-derived samples, totaling 105,487 multiview samples. Rendering eight 1024x1024 views per sample produces 843,896 images. Scene variation includes 7,953 ABO object meshes, Hair20K hairstyles, 863 HDR environment maps, and physically simulated clothing. Images and SMPL-X proxies come from the same rendering process, avoiding manual guesses about surface correspondences. These sample counts should not be interpreted as counts of distinct real identities.

The generator retains the pretrained SD 2.1 UNet backbone and trains only its attention modules and the T2I-Adapter. Text specifies the output modality and region, such as a full-body segmentation or UV map. T2I-Adapter residual features enforce pixel alignment, while frozen DINOv2 features provide pose and appearance information through image cross-attention. These conditions specify the desired output, its spatial alignment, and its visual context; they are not three independent mesh predictors.

Cross-modality attention concatenates UV and segmentation tokens to discourage disagreement between surface coordinates and part labels. Multiview epipolar attention uses Plucker ray embeddings to exchange information along geometrically plausible correspondences. Thus, consistency is encouraged during proxy prediction rather than left entirely to downstream fitting. The formulation assumes perspective cameras without lens distortion and should not be mistaken for an uncalibrated image matcher.

Synthetic data does not mean that every pose source is independent of the evaluation datasets. The authors use training-set motions or poses from MPI-3DHP and MoYo for some renders and state that test poses are excluded. The precise claim is that the proxy generator is not trained on real evaluation images, not that it has never used any pose from those datasets. One model is shared across evaluations without dataset-specific fine-tuning.

2. Hand refinement: turn small hands into additional views of the same network

Hands usually occupy less than 1% of full-body image pixels, leaving insufficient detail for fingers even when the overall pose is reasonable. The first pass generates full-body proxies and uses hand segmentation to create enlarged left- and right-hand crops. The second pass processes these crops together with body images using the same generator instead of a separately trained hand model.

Cropping leaves camera extrinsics unchanged but requires updating focal length and principal point so that new pixel coordinates retain the original rays. A hand-specific prompt switches to finer part labels. The text describes 12 parts as two palms and ten fingers; this should not be interpreted as ten fingers on a single hand. Cross-view attention connects enlarged local detail with full-body context, reducing the orientation or left-right ambiguity that isolated hand prediction could introduce.

Training fixes four view slots per sample, randomly using two to four full-body views and filling the remaining slots with hand crops from those cameras. Default inference uses four body views plus a left- and right-hand crop from each, yielding twelve proxy views. These are additional crops, not eight additional physical cameras, and no retraining is needed for the changed view count.

3. Sampling and reliability estimation: make diffusion stochasticity useful for fitting

Occlusion boundaries and left-right limb ambiguity can produce incorrect but visually confident correspondences in a single prediction. The method draws \(K\) stochastic proxies per view, with \(K=3\) by default. UV coordinates are aggregated by the pixel-wise median, and UV uncertainty is the mean sample variance over three channels. Segmentation RGB values are first quantized to the nearest valid part color and then aggregated by majority voting. If no label receives more than half of the votes, segmentation uncertainty is set to one.

The following notation reconstructs Eqs. (3)-(5) from their prose definitions. Formula extraction in the cache is damaged; this reconstruction does not introduce an additional normalization or variance-scaling step. Refer to the original paper and author implementation for exact implementation details.

\[ U_v^{\mathrm{uv}}(x)=\frac{1}{3}\sum_{c=1}^{3}\operatorname{Var}_{k=1..K}\left[P_{k,v}^{\mathrm{uv},(c)}(x)\right],\qquad U_v^{\mathrm{seg}}(x)=\begin{cases}1,&n_{\max}(x)\le K/2,\\2\left(1-\frac{n_{\max}(x)}{K}\right),&n_{\max}(x)>K/2,\end{cases}\qquad W_v(x)=\left(1-U_v^{\mathrm{uv}}(x)\right)\left(1-U_v^{\mathrm{seg}}(x)\right). \]

Here, \(n_{\max}(x)\) is the vote count of the most frequent label at a pixel. Disagreement in labels or unstable UV coordinates reduces that pixel's contribution, allowing fitting to rely on more stable views. It cannot correct a systematic error shared by every sample. Increasing \(K\) spends inference compute without updating network weights; when \(K=1\), the paper explicitly sets every weight to one.

4. Weighted geometric fitting: constrain one body through projected surface points

Each foreground pixel's part label and UV identify an SMPL-X surface point. Its 3D position changes as body shape and pose change; optimization brings its projection under the known camera close to the original image pixel. All views constrain the same body, so a reliable view from the other side can compensate for missing information under occlusion.

Following Eq. (6), the objective sums reliability-weighted squared reprojection distances rather than directly matching RGB colors in generated proxy images:

\[ \mathcal{L}_{\mathrm{reproj}}=\sum_v\sum_{x\in\mathrm{fg}(v)}W_v(x)\,d_v(x)^2. \]

Here, \(d_v(x)\) is the 2D Euclidean distance between the projected surface point and pixel \(x\). Body and hand poses are optimized in axis-angle space, with a one-step denoising pose prior from DPoser-X. Shape has no explicit regularization. Fitting optimizes shape, pose, translation, and global scale, but not facial expression; using SMPL-X does not imply that facial expression is recovered.

A Worked Example

Consider four cameras observing a person whose arm is partly occluded. The first pass produces four body proxies, from which left and right hands are cropped to create eight additional hand views. The second pass predicts twelve proxy sets and repeats sampling with \(K=3\). If the three segmentation labels at a leg pixel all differ, \(n_{\max}=1\), segmentation uncertainty is one, and its fitting weight is zero. If two votes agree, segmentation uncertainty is \(2/3\), and the final weight also depends on UV reliability.

This is an illustrative application of the stated rules, not an additional experiment. Fitting can still invert a joint in depth because projection agreement does not uniquely determine its 3D orientation, and the pose prior does not guarantee a global optimum.

Loss & Training

Proxy training uses standard diffusion noise prediction: encode a ground-truth proxy with the VAE, add noise, and predict that noise with the conditioned network. Training uses Adam, learning rate \(5\times10^{-5}\), batch size 2, and 30 epochs, taking about 36 hours on four RTX 5090 GPUs. The VAE decoder is separately fine-tuned for 100K iterations at learning rate \(10^{-6}\) and batch size 8, taking about four hours on the same hardware, to reduce quantization artifacts in numerically precise proxies.

Fitting also uses Adam, first solving global placement, then body pose and shape, and finally hand articulation. A stage advances when the relative loss decrease between consecutive iterations falls below its threshold; the stated threshold range is 1% to 10%. The cached Eq. (7) does not preserve the prior coefficient completely, so no value is invented here. Supplementary Algorithm 1 is absent from this cache, and its pixel-to-surface interpolation details cannot be reconstructed reliably from the available text.

On one RTX 5090, initial body proxies take about 3 seconds, twelve-view generation about 15 seconds per sample, and fitting with hands about 60 seconds. Summing these components for default \(K=3\) gives approximately \(3+3\times15+60=108\) seconds. This is a component-based estimate, not a separately measured end-to-end result. With hand refinement and repeated sampling disabled, the paper reports approximately 53 seconds.

Key Experimental Results

Main Results

MPJPE measures mean joint position error, and MPVPE measures mean vertex position error. PA variants apply Procrustes alignment; every value below is in mm, with lower being better. Section 4.3 fixes four physical views: 3DHP uses S8 and cameras 0/2/7/8; BEHAVE uses all four test-set views; RICH uses the first four valid test-set views; MoYo uses validation views 1/3/4/5; and 4D-DRESS uses its full dataset with all four views. Sampling uses every fifth frame except for 4D-DRESS, which uses every fiftieth frame. The partial setting adds random crops.

The following selection from original Table 1 compares the full model with EasyMoCap, another fitting-based system. Reductions are differences between their MPVPE values, not claims of superiority over every method.

Dataset/setting	EasyMoCap MPVPE	DiffProxy MPVPE	Reduction	DiffProxy MPJPE
3DHP	51.5	50.9	0.6	41.1
RICH	60.3	35.1	25.2	33.7
BEHAVE	40.4	39.9	0.5	32.7
MoYo	51.6	38.8	12.8	32.4
4D-DRESS	35.3	25.6	9.7	20.5
4D-DRESS partial	119.4	43.1	76.3	31.6

Single-view baselines are evaluated camera by camera after root alignment, and the single view with the lowest MPVPE is reported per dataset. This is not average performance on a randomly selected image. Stars in original Table 1 also indicate methods trained on the corresponding dataset. In particular, EasyMoCap's BEHAVE MPJPE is 29.6 versus 32.7 for DiffProxy, and HeatFormer's 3DHP PA-MPJPE is 29.2 versus 32.8 for DiffProxy.

Ablation Study

The following results come from original Table 2 on BEHAVE. The last column is relative to full-model MPVPE 39.9. Independent removal effects cannot be added together to predict a combined ablation.

Config	PA-MPJPE	MPJPE	PA-MPVPE	MPVPE	MPVPE increase
Full model	22.8	32.7	31.3	39.9	0.0
Without DINOv2	37.3	44.3	53.7	60.2	20.3
Without T2I-Adapter	28.5	41.5	36.7	48.4	8.5
Without text attention	26.5	35.2	35.3	43.3	3.4
Without epipolar attention	25.7	33.7	35.0	41.9	2.0
Without cross-modality attention	26.3	33.8	35.4	41.9	2.0

Original Table 1 also shows RICH MPVPE decreasing from 46.5 for the base configuration to 36.2 with hand refinement and 35.1 with repeated sampling. However, the corresponding 3DHP values are 50.5, 51.5, and 50.9, so hand refinement does not improve this metric. The prose claim of consistent improvement should not be extended to every reported measure.

Original Table 3 tests proxy sources with the same fitting pipeline; the following results are on MoYo:

Proxy source	PA-MPJPE	MPJPE	PA-MPVPE	MPVPE
DensePose	67.6	101.1	90.4	127.8
DensePose fine-tuned on synthetic data for 60 epochs	66.5	88.8	84.5	111.4
DiffProxy	28.3	32.4	35.2	38.8

Key Findings

• Removing DINOv2 produces the largest degradation, supporting the importance of pose and appearance priors for correspondence prediction. This does not establish that the diffusion backbone alone explains all gains.
• Moving from full 4D-DRESS observations to partial crops changes MPVPE from 25.6 to 43.1 for DiffProxy and from 35.3 to 119.4 for EasyMoCap. Dense constraints are more robust to partial visibility, but errors still increase substantially.
• Without ground-truth cameras, the VGGT-initialized variant with joint camera optimization reports only PA-MPJPE 25.9 and PA-MPVPE 37.3 in Table 2. Unreported non-PA errors must not be filled with zeros or compared directly with the full model's non-PA metrics.

Highlights & Insights

• The intermediate proxy is simultaneously visualizable, supervised, and usable for geometric optimization. Errors can be localized to segmentation, correspondence, or fitting rather than remaining entangled in parameter regression.
• Updating intrinsics allows hand crops to enter the existing multiview network while preserving geometry. Applications to other fine-grained reconstruction tasks must retain this crop-coordinate transformation.
• Stochastic sampling does more than average predictions: it limits the influence of disputed local evidence on the final mesh. Segmentation voting and UV variance capture categorical ambiguity and continuous-coordinate instability, respectively.

Limitations & Future Work

• Figure 10 shows a fitting local optimum with an inverted 3D joint despite correct 2D correspondences. Triangulated multiview correspondences or complementary 3D constraints may help; extra iterations of the same 2D loss are not a guaranteed fix.
• Inference takes tens of seconds to roughly a hundred seconds, making direct real-time capture impractical. The authors suggest regression-based initialization, diffusion distillation, or consistency models for acceleration.
• Single-view recovery remains ambiguous in depth, and the current system targets one person. Multihuman extension requires cross-view identity association, not simply repeated application of a single-person model.
• Reader observation: evaluation supports transfer without real-image fine-tuning, but some training pose sources relate to evaluation datasets, so split details matter. The DensePose comparison also does not isolate all effects of pretraining scale, architecture, and training budget.
• Reader observation: sample agreement does not guarantee correctness, and shared mislabeling can receive high weights. Confidence calibration and shared biases under clothing occlusion need further evaluation. The output is a parametric body mesh, not a complete reconstructed clothing surface.

Related Work & Insights

• Versus EasyMoCap: both explicitly fit a body model, but use sparse joints versus dense surface evidence. DiffProxy is more robust under difficult occlusion, although it does not beat EasyMoCap on BEHAVE joint error.
• Versus DensePose / DenseRaC: dense correspondence fitting is not new; the contribution is improving proxy quality through precise synthetic annotations, diffusion visual priors, and cross-view interaction.
• Connections to Marigold / GenPercept / SPAD: the first two establish diffusion backbones for dense perception, while SPAD motivates epipolar-constrained attention. DiffProxy applies these capabilities to human surface correspondences and downstream fitting.
• Versus Look Ma, No Markers: both use synthetic training and reprojection fitting, but that method predicts fixed dense 2D landmarks rather than per-pixel correspondences. Its unavailable model prevents a direct comparison in this paper, so superiority is not established.

Rating

• Novelty: 4/5. Dense proxies are established, but their combination with diffusion generation, local refinement, and uncertainty-weighted fitting is concrete and effective.
• Experimental Thoroughness: 4/5. Five real datasets, six settings, and module ablations provide substantial coverage, with matched-budget proxy comparisons and fuller camera-error analysis still missing.
• Writing Quality: 4/5. The pipeline and failure cases are clear, but broad superiority claims require table-level qualification, and cached equations suffer extraction damage.
• Value: 4/5. Useful for high-precision offline human reconstruction and annotation, with inference cost limiting real-time deployment.