Monocular Models are Strong Learners for Multi-View Human Mesh Recovery

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Human Mesh Recovery, Multi-view Reconstruction, Test-time Adaptation, Monocular Priors, Anatomical Landmarks

TL;DR

MonoMV presents a training-free multi-view human mesh recovery framework that leverages strong priors from pretrained single-view HMR models, initializing a robust "Virtual View" and refining it via test-time adaptation guided by anatomical landmarks, cross-view consistency, and monocular prior regularization to achieve SOTA accuracy under both calibrated and calibration-free setups.

Background & Motivation

Multi-view human mesh recovery (HMR), aiming to estimate accurate 3D human pose and shape from synchronized multi-camera streams, serves as an essential building block for fine-grained behavioral analysis, human digitization, and immersive AR/VR experiences. Compared with single-view estimators that fundamentally struggle with depth ambiguity, severe occlusion, and viewpoint-dependent uncertainty, multi-view approaches theoretically offer complementary spatial observations to substantially enhance geometric fidelity and reconstruction robustness. However, existing multi-view HMR paradigms face a persistent dilemma. Classical geometric approaches relying on voxelization and triangulation demand rigid, error-prone camera extrinsic calibration and suffer from irreversible voxel quantization errors. On the other hand, recent learning-based multi-view frameworks (such as U-HMR and MVP) achieve competitive performance on specific training distributions, yet their learned multi-view fusion modules inevitably memorize dataset-specific geometric priors and 2D-to-3D correspondences due to the severe scarcity of camera configurations in available multi-view datasets (e.g., Human3.6M, MPI-INF-3DHP). Consequently, when deployed to novel, unseen camera topologies, their performance collapses dramatically.

Concurrently, single-view HMR has experienced rapid progress fueled by massive-scale real and synthetic human datasets (e.g., AMASS, BEDLAM, COCO, 4DHumans). State-of-the-art monocular models such as HMR2.0, TokenHMR, and CameraHMR have learned highly expressive, versatile 3D human pose and shape priors across diverse camera viewpoints. Yet, standard practice has largely treated multi-view reconstruction as an isolated problem requiring dedicated multi-view network training, underestimating the potential of monocular priors to resolve multi-view spatial ambiguities through explicit test-time geometric optimization.

Given that purely data-driven multi-view fusion modules fail to generalize beyond their training layouts, while monocular models possess rich human body priors learned from abundant data, an important question arises: Can we construct a multi-view HMR framework relying exclusively on single-view models that generalizes robustly across arbitrary, uncalibrated camera setups without any multi-view training? The core idea is to bypass multi-view training entirely by synthesizing monocular predictions into a unified "Virtual View" initialization and executing lightweight test-time adaptation (TTA) that alternates updates between single-view latent tokens and virtual-view SMPL parameters under 2D anatomical landmark guidance, cross-view consistency, and prior regularization.

Method

Overall Architecture

The MonoMV pipeline operates in two sequential stages: Virtual View Initialization and Test-Time Adaptation (TTA). In the initialization stage, \(N\) synchronized RGB camera views are passed through a frozen single-view Transformer HMR backbone to generate initial per-view SMPL parameters and latent tokens. A standard-deviation filtering mechanism is applied across views to prune erratic joint rotations before weighted aggregation, establishing a robust global "Virtual View" SMPL parameter set. In the adaptation stage, the framework iteratively refines the single-view latent tokens and the virtual-view SMPL parameters. This test-time optimization is driven by three cooperative losses: an anatomical 2D reprojection loss penalizing both joint keypoints and surface landmarks to constrain limb twist rotations, a cross-view geometric consistency loss enforcing consensus across views and against the virtual view, and a monocular prior regularization loss preventing divergence from the reliable single-view manifold.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Synchronized Multi-View RGB Images (N Views)"] --> B["Frozen Monocular HMR Backbone Forwarding<br/>Extract Predictions and Latent SMPL Tokens"]
    B --> C["Robust Statistical Filtered Virtual View Initialization<br/>Prune Outlier Joint Rotations and Aggregate"]
    C --> D["Monocular Learned SMPL Tokens Update<br/>Backpropagate Exclusively Through Lightweight MLP Head"]
    D --> E["Anatomical Landmark and Joint Reprojection Guidance<br/>44 Skeleton Joints + 35 Surface Landmarks for Twist"]
    E --> F["Cross-View Geometric Consistency Constraint<br/>Pairwise or Star-Graph Consensus Alignment"]
    F --> G["Monocular Prior Regularization<br/>Anchor Optimization to Initial High-Quality Latents"]
    G --> H["Converged Global Virtual-View 3D Human Mesh"]

Key Designs

1. Robust Statistical Filtered Virtual View Initialization: Establishing an Outlier-Resilient Geometric Anchor

Directly averaging monocular SMPL outputs across views often leads to degraded initial poses because individual monocular estimates are prone to monocular depth ambiguities or severe local occlusions. To establish a clean starting point, MonoMV employs a \(1-\delta\) standard-deviation filtering mechanism. For each joint rotation \(k\) parameterized in continuous 6D space \(\boldsymbol{\theta}_i^{(k)}\), the Euclidean distance \(d_i^{(k)}\) to the empirical mean across all \(N\) views is computed. The standard deviation of these distances \(\operatorname{Std}(d_1^{(k)}, \dots, d_N^{(k)})\) defines an adaptive rejection threshold, isolating the reliable subset \(\mathcal{S}^{(k)}\):

\[\mathcal{S}^{(k)} = \left\{ \boldsymbol{\theta}_i^{(k)} \;\middle|\; d_i^{(k)} \le \operatorname{Std}(d_1^{(k)}, \dots, d_N^{(k)}) \right\}\]

The virtual-view initial pose \(\boldsymbol{\theta}_0^{(k)}\) is calculated as the mean of \(\mathcal{S}^{(k)}\), while the shape parameters \(\boldsymbol{\beta}_0\) are averaged over all views. For global orientation (root joint rotation), when camera calibration is available, per-view root rotations are transformed into the world coordinate frame via camera extrinsics prior to filtering; in calibration-free mode, global orientations remain view-specific and are excluded from multi-view averaging, while relative joint rotations are shared and averaged. This design guarantees a stable geometric core across diverse setups.

2. Dual Latent-Explicit Optimizable Variables: Balancing High Expressiveness with Computational Efficiency

A central question in test-time adaptation is selecting which parameters to optimize. Updating the full Vision Transformer encoder or decoder weights is computationally prohibitive and risks overwriting the rich prior learned from large-scale single-view data. Conversely, optimizing only the low-dimensional explicit SMPL parameters (72 pose and 10 shape parameters) fails to leverage the image-conditioned latent representations and frequently gets trapped in poor local minima caused by noisy 2D detections. MonoMV resolves this trade-off by adopting a dual-variable scheme: on the single-view side, it optimizes the Learned SMPL Tokens produced by the transformer decoder. These tokens encode concentrated, image-conditioned human representations; during optimization, gradients only pass through a single linear MLP projection head without propagating through the ViT backbone. On the multi-view side, the virtual-view pose, orientation, and shape parameters are updated explicitly. Alternating updates between these complementary representations preserves backbone priors while providing adequate flexibility for multi-view consensus.

3. Joint Anatomical Landmark and 2D Keypoint Reprojection Guidance: Disambiguating Limb Twist Rotations

Standard 2D skeleton detectors only output joint centers, leaving axial limb rotations (such as forearm pronation or wrist twist) unconstrained because twisting a limb barely alters 3D joint locations. To overcome this limitation, MonoMV incorporates the SynthPose detector to extract 44 skeleton keypoints and 35 anatomical surface landmarks with confidence exceeding 0.9 for each camera view. The 35 surface landmarks are indexed directly from the corresponding SMPL mesh vertices \(\mathcal{M}_i\), projected onto the image plane via camera model \(\Pi_i\), and supervised against detector outputs with confidence weights \(D_i\):

\[\mathcal{L}_{\text{2D}} = \sum_{k \in \mathcal{K}_{\text{kp}}} D_i^{(k)} \| P_{i,r}^{(k)} - \hat{P}_i^{(k)} \|_2 + \sum_{m \in \mathcal{K}_{\text{ana}}} D_i^{(m)} \| P_{i,a}^{(m)} - \hat{P}_i^{(m)} \|_2\]

Because anatomical surface points exhibit prominent 2D displacements under subtle limb rotations, this combined objective provides sensitive gradient signals that correct rotational misalignments that joint-only supervision overlooks.

4. Cross-View Geometric Consistency and Monocular Prior Regularization: Enforcing Multi-View Alignment Without Manifold Drifting

To reconcile predictions into a coherent 3D body across different vantage points, a cross-view consistency loss \(\mathcal{L}_{\text{con}}\) penalizes pairwise discrepancies in SMPL pose parameters \(\boldsymbol{\theta}\), shape parameters \(\boldsymbol{\beta}\), and full vertex meshes \(\mathcal{M}\). When refining the virtual view, consistency is enforced between the virtual parameters and all camera views; when refining monocular tokens, pairwise consistency among camera views is enforced (computable in \(O(N)\) complexity using a star graph). Furthermore, to prevent the adaptation from over-accommodating noisy 2D detections and drifting into anatomically invalid configurations, a prior regularization loss \(\mathcal{L}_{\text{reg}}\) anchors the parameters to the initial step-0 estimates:

\[\mathcal{L}_{\text{reg}} = \sum_{i=1}^N \left( \|\boldsymbol{\theta}_{i,t} - \boldsymbol{\theta}_{i,0}\|_2 + \|\boldsymbol{\beta}_{i,t} - \boldsymbol{\beta}_{i,0}\|_2 + \|\mathcal{M}_{i,t} - \mathcal{M}_{i,0}\|_2 \right)\]

This regularizer acts as an elastic constraint ensuring that the optimized output retains the plausible body manifold synthesized by the monocular pretraining.

Loss & Training

During the test phase, 200 optimization steps are executed per multi-view sample. At step \(t\), parameter updates proceed according to:

\[\Theta_{i,t+1} = \Theta_{i,t} - \eta \nabla_{\Theta_{i,t}} (\mathcal{L}_{\text{2D}} + \lambda \mathcal{L}_{\text{con}} + \gamma \mathcal{L}_{\text{reg}})\]

where \(\eta\) is the learning rate, and \(\lambda\) and \(\gamma\) weight cross-view consistency and prior regularization, respectively. The single-view latent tokens and the virtual-view parameters are alternately updated using gradients computed on the fly, requiring zero multi-view training data or offline model fine-tuning.

Key Experimental Results

Main Results

Quantitative comparisons on the Human3.6M and MPI-INF-3DHP benchmarks against state-of-the-art single-view, calibration-free multi-view, and calibrated multi-view methods are summarized below.

Dataset / Benchmark Protocol	Method	Backbone / Paradigm	MPJPE (mm) ↓	PA-MPJPE (mm) ↓	Comparison against Prior SOTA
Human3.6M (Single-View)	HMR2.0 (b)	Monocular Pretrained	50.0	32.4	Baseline reference
Human3.6M (Single-View)	PostureHMR	Monocular Pretrained	44.5	31.0	Baseline reference
Human3.6M (Calibration-Free)	PaFF	Neural Optimization	44.8	28.2	Multi-view baseline
Human3.6M (Calibration-Free)	HeatFormer	Multi-View Supervised	42.5	25.8	Previous uncalibrated SOTA
Human3.6M (Calibration-Free)	U-HMR	Multi-View Supervised	31.0	22.8	Memorizes training setup
Human3.6M (Calibration-Free)	Ours	TokenHMR	44.3	23.9	-1.9 mm PA-MPJPE
Human3.6M (Calibration-Free)	Ours	HMR2.0	45.2	23.0	-2.8 mm PA-MPJPE
Human3.6M (Calibration-Free)	Ours	CameraHMR	32.7	22.0	-3.8 mm PA-MPJPE (New SOTA)
Human3.6M (Calibrated)	MV-SPIN	Volumetric / Optimization	49.8	35.4	Classical multi-view
Human3.6M (Calibrated)	PaFF	Pixel-Aligned Feedback	33.0	26.9	Neural optimization
Human3.6M (Calibrated)	HeatFormer	Transformer Optimizer	29.5	22.4	Previous calibrated SOTA
Human3.6M (Calibrated)	Ours	TokenHMR	32.6	23.2	Training-free generalization
Human3.6M (Calibrated)	Ours	HMR2.0	31.6	21.5	-0.9 mm PA-MPJPE
Human3.6M (Calibrated)	Ours	CameraHMR	26.9	20.6	-2.6 MPJPE / -1.8 PA-MPJPE

On the MPI-INF-3DHP dataset, Ours (CameraHMR) achieves 39.0 mm MPJPE, 99.9% PCK, and 83.8 AUC, outperforming HeatFormer (39.8 mm / 99.5% / 72.8). When evaluated under cross-camera generalization (evaluating on unseen cameras 1, 4, 5, 6 instead of 0, 2, 7, 8), HeatFormer degrades to 45.74 mm MPJPE and 68.93 AUC, whereas Ours maintains 43.71 mm MPJPE and 80.62 AUC (+11.69 AUC margin), verifying strong resilience to camera topology shifts.

Ablation Study

Ablation experiments conducted on Human3.6M using CameraHMR highlight the individual impact of each proposed module:

Ablation Dimension	Configuration	MPJPE (mm) ↓	PA-MPJPE (mm) ↓	Observation & Insight
Virtual View & TTA	w/o Virtual View + w/ TTA	27.8	22.7	Independent per-view refinement lacks a unified anchor
	w/ Virtual View + w/o TTA	44.2	29.9	Statistical initialization alone cannot resolve ambiguity
	Full Model (w/ Both)	26.9	20.6	Cooperative combination yields 17.3 mm MPJPE reduction
Initialization Strategy	Canonical T-pose	27.2	21.2	Uninformative prior slows down test-time convergence
	Arithmetic Average	27.1	20.7	Sensitive to outlier predictions and severe occlusions
	Standard-Deviation Weighted	26.9	20.6	Robust outlier filtering provides the cleanest initial state
TTA Loss Components	2D Keypoints Only (kp)	39.6	28.5	Lacks surface twist awareness and cross-view geometry
	Keypoints + Landmarks (kp+ana)	36.8	27.2	Anatomical landmarks resolve limb rotation ambiguity
	kp + ana + Cross-View Consistency	27.3	20.9	Cross-view geometric consensus yields the largest gain (-9.5 mm)
	Standard Guidance (+ Regularizer)	26.9	20.6	Regularizer prevents overfitting to 2D detector noise
Monocular Variable Choice	Explicit SMPL Parameters	58.2	32.2	Severe non-linearity causes trapping in poor local minima
	Full Decoder Weights	30.8	22.8	High dimensionality disrupts pretrained representations
	SMPL Query Tokens	29.7	20.9	Input queries lack abstracted image-conditioned features
	Learned SMPL Tokens	26.9	20.6	Ideal balance between expressiveness and efficiency

Key Findings

• Cross-View Consistency Drives the Primary Improvement: Adding cross-view geometric consistency on top of 2D reprojection reduces MPJPE from 36.8 mm to 27.3 mm, confirming that explicitly enforcing 3D geometric consensus is the decisive factor in eliminating monocular depth ambiguities.
• Optimizing Latent Tokens Vastly Outperforms Explicit Parameters: Directly optimizing explicit SMPL parameters yields a poor MPJPE of 58.2 mm due to non-convex optimization landscapes, whereas optimizing Learned SMPL Tokens achieves 26.9 mm, proving that the image-conditioned latent space acts as a natural regularizer.
• Geometry-Driven TTA Overcomes Dataset Bias: While fully supervised multi-view models degrade noticeably when evaluated across camera layouts, test-time optimization grounded in geometric and anatomical constraints generalizes reliably to unseen camera arrangements.

Highlights & Insights

• Decoupling Monocular Expressiveness from Multi-View Geometry: Rather than collecting scarce and expensive multi-view datasets to train specialized multi-view architectures, MonoMV repurposes foundational single-view models as generic body pose generators and leverages multi-view cues solely at inference time.
• Anatomical Surface Landmarks for Axial Twist Disambiguation: DensePose-derived surface landmarks provide vital 2D tangential motion cues under limb axial rotations, effectively addressing a classical blind spot of joint-only 2D supervision.
• Graceful Degradation Under Calibration-Free Setups: In the absence of camera extrinsics, the framework isolates global orientation to per-view variables while seamlessly sharing relative body pose and shape across views, maintaining state-of-the-art accuracy in uncalibrated scenarios.

Limitations & Future Work

• Inference Latency Overhead: Requiring 150–200 optimization steps per frame results in runtime latency that is substantially higher than feedforward networks, posing challenges for real-time interactive applications.
• Dependency on 2D Detector Accuracy: In extreme low-light or severe multi-person occlusion environments, false positives or missing detections from the 2D landmark detector can bias the gradient updates.
• Future Directions: Developing learnable neural optimizers to accelerate TTA convergence within 10–20 steps and extending the framework to temporal multi-view video sequences represent promising research avenues.

Related Work & Insights

• vs U-HMR / MVP: Feedforward multi-view networks absorb specific camera layouts during training and collapse under novel viewpoints; MonoMV is completely training-free and adapts dynamically to arbitrary camera positions.
• vs HeatFormer: HeatFormer relies on epipolar geometry via AdaFuse modules and requires multi-view training, resulting in reduced performance without calibration; MonoMV outperforms HeatFormer under both calibrated and calibration-free settings.
• vs PaFF: PaFF relies heavily on camera parameters to guide pixel-aligned feature feedback and scale recovery; MonoMV optimizes latent tokens and filtered virtual views, yielding a lighter and more resilient optimization trajectory.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Challenges the conventional wisdom of end-to-end multi-view training by demonstrating that monocular priors combined with TTA outperform supervised multi-view models]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across calibrated, calibration-free, cross-camera, and multi-backbone settings accompanied by deep ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, elegant methodology structure, and thorough analysis of optimization variables]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly generalizable paradigm for real-world markerless multi-view motion capture and human mesh recovery]