MuCHeR: Multi-Person Camera-Centric Human Detection, Mesh Recovery and Tracking

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/naver/multi-hmr2
Area: 3D Vision
Keywords: human mesh recovery, multi-person detection, camera-centric localization, DETR, human tracking

TL;DR

MuCHeR (Multi-HMR 2) is a simple and robust DETR-based one-stage framework for multi-person 3D human detection, mesh recovery, and tracking that significantly reduces training cost and enables heterogeneous 2D supervision via 2D-location Hungarian matching, achieves precise camera-centric metric 3D localization without ground-truth intrinsics through predicted vertical FOV, and achieves online multi-person tracking without any video training by distilling SAM2 memory encoder features.

Background & Motivation

Perceiving multi-person 3D human mesh recovery (HMR) from monocular RGB images is a cornerstone capability for robotics, human-robot interaction, virtual reality, and social scene understanding. Historically, the field has been dominated by two-stage "top-down" approaches: an off-the-shelf 2D object detector first detects bounding boxes for all persons in the scene, and a single-person HMR network subsequently regresses a 3D mesh for each individual crop independently. While conceptually straightforward, this paradigm suffers from inference latency that scales linearly with the crowd size, discards global inter-person spatial context and physical interactions, and relies on brittle heuristics to place individuals back into a coherent 3D scene coordinate system. To overcome these limitations, single-stage approaches like CenterNet-based detectors have emerged; however, when individuals closely interact or overlap within the same feature grid cell, they frequently encounter severe identity collisions and missed detections.

Recent efforts have turned to the DETR architecture to model global context and handle occlusions through learned queries. Nevertheless, existing DETR-based HMR pipelines, such as AiOS and SAT-HMR, face acute computational and representational roadblocks. Specifically, their bipartite matching procedures require passing all queries through a fully differentiable parametric body model to evaluate 3D joint distance matrices against ground truth. This causes peak GPU memory usage to balloon past 27GB and slows training step latency substantially, while strictly precluding the use of ubiquitous in-the-wild datasets annotated solely with 2D keypoints. Furthermore, prior approaches make rigid, unrealistic camera assumptions—such as a fixed 5000 mm focal length in AiOS or a constant 60° field of view in SAT-HMR—which induces dramatic metric depth estimation errors (Abs-PVE reaching tens of meters) whenever images deviate from the assumed camera geometry. In addition, relying on legacy adult-only SMPL templates forces models to hallucinate children as distant adults to reconcile visual scale, and extending HMR to video tracking has traditionally required dense, expensive video and motion sequence supervision.

To address these compounding bottlenecks, this paper re-evaluates the end-to-end design space across matching efficiency, camera geometry, morphological diversity, and temporal association. Core idea: build an end-to-end DETR framework around the inclusive Anny body model, accelerate training and unlock 2D supervision via 2D-only location Hungarian matching, achieve accurate metric 3D localization without ground-truth intrinsics by predicting camera field of view, and enable zero-video online tracking through distillation of SAM2 memory encoder features.

Method

Overall Architecture

MuCHeR follows a streamlined one-stage DETR architecture. Given an input RGB image cropped to diverse aspect ratios with its maximum dimension set to 768 pixels, a ViT-Large backbone initialized with DINOv3 extracts visual patch tokens alongside a global image-level [CLS] token. An MLP head regresses the camera vertical field of view (\(v\text{FOV}\)) directly from the [CLS] token to parameterize an intrinsic camera matrix for inverse perspective projection. The transformer decoder ingests 100 learned human queries that cross-attend to the image tokens across 8 blocks. The decoded queries branch into dedicated MLP prediction heads: detection confidence scores, 2D pixel locations and log-space depths (nearness), Anny body pose and shape parameters, and 4096-dimensional tracking features. During training, bipartite matching is performed strictly over 2D pixel coordinates and confidence scores, allowing unassigned queries to be discarded before any mesh generation. At inference time, high-confidence queries are inversely projected into the camera coordinate system using the predicted intrinsics, and their distilled tracking features are matched online across frames via KNN similarity and pelvis motion trajectory filtering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    IN["Input RGB Image<br/>Variable aspect ratio crop (max dim 768)"] --> ENC["ViT-Large Backbone<br/>Extracts patch tokens and CLS token"]
    ENC --> D1["Camera Intrinsics Prediction & Inverse Projection<br/>CLS token regresses vertical FOV for camera-centric unprojection"]
    ENC --> DEC["DETR Decoder<br/>100 learned queries interact via 8 cross-attention layers"]
    DEC --> D2["2D-Location Bipartite Matching<br/>Hungarian matching based solely on 2D pixel locations and scores"]
    D2 --> D3["Inclusive Anny Body Model & Adaptive Supervision<br/>Regresses 163 6D rotations and 6 shape parameters with multi-source loss"]
    DEC --> D4["SAM2 Memory Feature Distillation for Zero-Video Tracking<br/>Regresses 4096-d memory features combined with ridge regression tracking"]
    D1 & D3 & D4 --> OUT["Camera-Centric 3D Meshes & Tracklets<br/>Metric-scale multi-person scene reconstruction and identity association"]

Key Designs

1. Camera Intrinsics Prediction & Inverse Projection: accurate camera-centric metric localization from predicted field of view

Prior multi-person HMR methods assume a fixed focal length or rigid FOV, causing substantial metric localization errors when real-world camera intrinsics deviate from assumptions. MuCHeR directly resolves this by attaching a lightweight 3-layer MLP to the ViT encoder's global [CLS] token to regress the vertical field of view (\(v\text{FOV}\)), assuming a centered principal point and square pixels. For 3D root positioning, instead of directly regressing unstable absolute \((X, Y, Z)\) coordinates, the network predicts the 2D bounding center \(\hat{l}_i\) in pixel space and the depth in log-space (nearness \(n_i = \log(1/z_i)\)). The metric 3D position in camera space is recovered analytically via the inverse projection operator \(\Pi_K^{-1}\) using the predicted intrinsics \(K\). Furthermore, to ensure stable gradient backpropagation regardless of distance, the model introduces an angular reprojection loss over the continuous unit sphere: $\(\mathcal{L}_{\text{2D angle}} = \frac{1}{s_K} \angle\left(\Pi_K^{-1}(\hat{u}_i), \Pi_K^{-1}(u_i)\right)\)$ where \(s_K\) denotes the camera diagonal FOV angle. This formulation yields camera predictions that remain remarkably stable across video sequences (standard deviation of ~2.5°), enabling precise camera-centric placement without requiring ground-truth intrinsics at test time.

2. 2D-Location Bipartite Matching: lightweight assignment accelerating training and unlocking pure 2D supervision

Existing DETR-based HMR approaches perform Hungarian matching by forwarding all 100 queries through the parametric body model to compare full 3D joint positions against ground-truth meshes. This brute-force strategy incurs an exorbitant memory footprint of 27GB per step and restricts training exclusively to datasets with complete 3D mesh annotations. MuCHeR decouples matching from mesh generation by conducting bipartite Hungarian matching solely on predicted 2D pixel coordinates \(\hat{l}_i\) and classification confidence scores \(\hat{c}_i\): $\(\min_{x_{ij}} \sum_{i=1}^N \sum_{j=1}^M x_{ij} \left( \|\hat{l}_i - l_j\|_1 + F_C(\hat{c}_i) \right)\)$ where \(F_C(\hat{c}_i) = -\alpha(1-\hat{c}_i)^\gamma \log(\hat{c}_i)\) represents the standard Focal Loss (\(\alpha=0.25, \gamma=2\)). Negative queries are pruned immediately after matching, and only the matched subset (at most \(M\) queries) is passed forward through the differentiable Anny body model. This reduces training memory consumption by 74% (from 27GB down to 7GB) and accelerates per-step iteration time from 0.48s to 0.35s. Crucially, this mechanism enables end-to-end multi-task training on massive 2D keypoint datasets such as MS-COCO and OpenImages that lack any 3D annotations.

3. Inclusive Anny Body Model & Adaptive Supervision: modeling diverse human morphology while preserving 3D biomechanical priors

Standard SMPL models represent only adult proportions, causing models to mispredict children as distant adults to reconcile image-space scale. MuCHeR adopts the open-source Anny parametric body model, which covers human morphology across all age groups (from infants to the elderly) and diverse body types using 163 joint rotations (in continuous 6D representation) and 6 interpretable shape parameters. To harvest signals across heterogeneous sources without negative transfer, an adaptive multi-tier supervision strategy is deployed: synthetic datasets (Anny-One, BEDLAM) supervise all heads (camera, detection, depth, mesh, joints); real images with pseudo-GT meshes (COCO, MPII, AIC fitted by CameraHMR) supervise 3D body pose and shape while masking detection penalties on unannotated persons; and pure 2D datasets supervise detection and 2D keypoint reprojection. To eliminate the pathological tendency where occluded lower bodies collapse into "sitting poses" under sparse 2D supervision, the fine-tuned model (MuCHeR.b) incorporates an L2 consistency regularization loss on 3D pose parameters against the 3D-only pretrained base model (MuCHeR.a), preserving rigid anatomical priors while elevating 2D detection recall.

4. SAM2 Memory Feature Distillation for Zero-Video Tracking: zero-shot temporal association from static image representations

Multi-person 3D tracking in video typically requires supervised training on dense video sequences or vulnerable hand-crafted appearance texture maps. MuCHeR leverages the rich spatio-temporal memory representation of the SAM2 foundation model using only static images. For any training image with ground-truth segmentation masks, the image is encoded by the SAM2 image encoder, and the resulting feature map is passed with the human segmentation mask into the SAM2 memory encoder to produce a \(64 \times 64 \times 64\) memory tensor. This representation is compressed via \(8 \times 8\) average pooling and flattened into a 4096-dimensional descriptor regressed by the DETR tracking head. During online video inference, each frame is processed independently: extracted query features are queried against an active memory bank (past 50 frames) using KNN (\(K=35\)) with softmax temperature scaling, while active pelvis trajectories (past 8 frames) are extrapolated via ridge regression. Fusing appearance memory similarity with smooth 3D motion priors enables continuous online tracking and robust re-identification across occlusions without having seen a single video during training.

Loss & Training

The overall training loss combines multi-task objectives: $\(\mathcal{L} = \lambda_{\text{cls}} \mathcal{L}_{\text{cls}} + \lambda_{\text{loc}} \mathcal{L}_{\text{loc}} + \lambda_{\text{depth}} \mathcal{L}_{\text{depth}} + \lambda_{\text{cam}} \mathcal{L}_{\text{cam}} + \lambda_{\text{pose}} \mathcal{L}_{\text{pose}} + \lambda_{\text{mesh}} \mathcal{L}_{\text{mesh}} + \lambda_{\text{2D}} \mathcal{L}_{\text{2D angle}} + \lambda_{\text{track}} \mathcal{L}_{\text{track}}\)$ where \(\mathcal{L}_{\text{cam}}\) employs an asymmetric penalty to avoid underestimating focal lengths, and \(\mathcal{L}_{\text{track}}\) denotes the L1 distillation loss against SAM2 memory vectors. The backbone is initialized with DINOv3 and trained with Adam on a single GPU in approximately one week. The training pipeline consists of pretraining on 3D annotations (MuCHeR.a), followed by regularized joint fine-tuning with large-scale 2D annotations (MuCHeR.b).

Key Experimental Results

Main Results

Evaluations across in-the-wild HMR benchmarks demonstrate state-of-the-art pelvis-centered accuracy and unprecedented metric camera-centric localization (Table 1), robust multi-person association under close interaction (Table 2), and competitive zero-video tracking performance (Table 2).

Table 1: Evaluation on in-the-wild HMR benchmarks (3DPW and EMDB)

Method	Camera	3DPW PA-MPJPE (mm) ↓	3DPW PVE (mm) ↓	3DPW TA-PVE (mm) ↓	EMDB PA-MPJPE (mm) ↓	EMDB PVE (mm) ↓	EMDB Abs-PVE (mm) ↓
AiOS	Fixed (5000mm)	45.0	90.9	774.3	63.3	108.1	11962.1
SAT-HMR	Fixed (60°)	52.7	94.5	174.5	71.0	126.7	1093.0
Multi-HMR	GT	46.9	88.8	187.2	48.5	87.1	168.1
MuCHeR.a	Predicted	40.5	80.1	170.1	48.3	78.7	273.6
MuCHeR.b	Predicted	41.8	78.1	161.9	52.5	83.6	374.8
MuCHeR.a	GT	40.2	79.9	149.8	48.3	78.7	129.4
MuCHeR.b	GT	41.6	77.7	145.6	52.5	83.6	131.7

Table 2: Close interaction benchmarks (Hi4D, Harmony4D) and video tracking (PoseTrack21)

Benchmark	Method	Camera / Supervision	MPJPE (mm) ↓	Pair-PA-MPJPE (mm) ↓	TA-PVE (mm) ↓	Detection F1 / Recall ↑
Hi4D	AiOS	Fixed	72.9	239.0	694.4	98% (F1)
	SAT-HMR	Fixed	88.2	85.5	132.4	100% (F1)
	Multi-HMR	GT	69.2	82.2	116.2	98% (F1)
	MuCHeR.a	Predicted	61.7	79.5	114.9	100% (F1)
	MuCHeR.a	GT	61.7	78.3	113.1	100% (F1)
Harmony4D	AiOS	Fixed	130.5	304.8	3454.8	91% (Recall)
	SAT-HMR	Fixed	140.3	144.0	676.3	97% (Recall)
	Multi-HMR	GT	146.0	140.8	388.1	93% (Recall)
	MuCHeR.a	Predicted	131.8	99.8	231.2	97% (Recall)
	MuCHeR.a	GT	128.4	96.2	233.6	97% (Recall)
PoseTrack21	Humans4D	Still Images (Two-stage)	-	-	-	IDF1: 72.5% / MOTA: 60.2%
	MuCHeR.b	Still Images (Zero-video supervision)	-	-	-	IDF1: 75.2% / MOTA: 65.5%
	CoMotion	Video Sequence Supervision	-	-	-	IDF1: 79.5% / MOTA: 70.1%

Ablation Study

Controlled ablations highlight the specific impacts of matching formulations, data supervision combinations, aspect ratio augmentations, and camera learning (Table 3).

Table 3: Ablations on matching strategy, training data mix, and camera estimation

Dimension	Configuration	3DPW PVE (mm) ↓	3DPW PA-MPJPE (mm) ↓	Hi4D Pair-PA-MPJPE (mm) ↓	COCO Keypoint AP ↑	COCO Keypoint AR ↑
Matching Strategy	3D Joint Matching (J3D)	104.5	55.1	84.8	8.6	16.7
	3D Location Matching	93.0	46.8	76.4	17.0	30.4
	2D Location Matching (Ours)	90.5	45.6	77.8	21.9	29.4
Training Data	Synthetic only (S)	93.6	48.2	64.0	11.9	24.7
	Real Pseudo-GT only (R)	88.7	45.7	75.7	21.3	26.6
	Synthetic + Real Pseudo-GT (S+R)	90.5	45.6	77.8	21.9	29.4
	Naive S+R+2D (unregularized)	91.6	46.9	84.2	16.8	32.9
	Regularized Fine-tuning (MuCHeR.b)	-	-	-	50.7	56.2
Camera Settings	Train GT / Test GT	90.5 (Abs: 148.0)	45.6	77.8	-	-
	Train GT / Test Predicted	92.4 (Abs: 609.8)	-	91.0	-	-
	Train Predicted / Test Predicted	86.2 (Abs: 577.3)	-	72.7	19.6	31.0

Key Findings

• 2D location matching outperforms costly 3D matching: Under an equivalent compute budget, 2D location matching achieves a PVE of 90.5 mm on 3DPW compared to 104.5 mm for full 3D joint matching. Eliminating body model passes during bipartite matching conserves memory and accelerates convergence.
• Learned camera parameters break metric localization barriers: On EMDB, AiOS exhibits an extreme absolute depth error (Abs-PVE) of 11,962.1 mm (nearly 12 meters) due to its 5000 mm focal length heuristic. MuCHeR reduces this error to 273.6 mm with predicted FOV, matching or surpassing specialized camera calibration methods.
• Regularization prevents 2D-induced pose collapse: Directly injecting 2D keypoint datasets without regularization degrades 3D accuracy due to ambiguous sitting-pose hallucinations under lower-body occlusion. Distilling 3D pose constraints from the 3D-only base model (MuCHeR.a) boosts COCO keypoint AP from 34.0 to 50.7 without compromising 3D biomechanics.

Highlights & Insights

• Decoupled 2D assignment unlocks scalability: Demonstrating that 2D location matching suffices for complex 3D mesh queries is an elegant architectural insight, simultaneously cutting step training time by 27%, slashing memory by 74%, and removing the prerequisite of 3D annotations during candidate matching.
• Cross-task foundation model feature distillation: Siphoning temporal tracking features from the memory encoder of a static image/video segmentation model (SAM2) provides a powerful blueprint for borrowing zero-shot temporal capabilities without needing dedicated video capture datasets.
• Morphological inclusiveness restores metric fidelity: Incorporating the unified Anny model across infants, children, and adults eliminates the fundamental scale-depth ambiguity of prior adult-centric body models, providing an honest baseline for real-world interactive human capture.

Limitations & Future Work

• Unmodeled optical distortion in wide-angle cameras: The formulation assumes an idealized pinhole model without radial or tangential lens distortion, causing registration errors under extreme fisheye or selfie perspectives.
• Absence of explicit physical inter-penetration constraints: While achieving top-tier Pair-PA-MPJPE scores during tight interactions, the framework lacks explicit mesh-to-mesh collision penalty layers, occasionally resulting in minor mesh interpenetrations.
• Convergence overhead of standard DETR: Even though single-GPU training in one week is efficient compared to prior DETR-based HMR systems, convergence remains slower than CNN architectures. Future iterations could integrate modern DETR convergence accelerators such as Group DETR or DINO matching.

Related Work & Insights

• vs AiOS (ECCV 2022): AiOS established DETR for multi-person HMR but assumed a fixed 5000 mm camera, required multi-stage human and joint tokens, and computed heavy 3D joint matching; MuCHeR unifies detection, mesh regression, and tracking with single-stage 2D matching and active FOV prediction, reducing absolute localization errors by two orders of magnitude.
• vs Multi-HMR (ECCV 2024): Multi-HMR used a CenterNet backbone that struggles when subjects' centers coincide within the same feature patch; MuCHeR's query-based transformer decoder naturally isolates overlapping humans, achieving a 100% detection F1-score on Hi4D.
• vs Humans4D (ICCV 2023): Humans4D combined two-stage mesh fitting with manual texture bank matching for tracking; MuCHeR operates end-to-end and leverages distilled SAM2 memory representations, outperforming Humans4D on PoseTrack21 (MOTA 65.5% vs 60.2%) without using video sequences during training.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Pioneering 2D decoupled matching for DETR HMR, integrated vertical FOV prediction, and zero-video tracking via SAM2 memory distillation.]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across in-the-wild HMR, dense interactions, camera-centric translation, 2D keypoints, and video tracking.]
• Writing Quality: ⭐⭐⭐⭐⭐ [Concise narrative with clear articulation of prior camera and matching limitations, supported by clean figures and tables.]
• Value: ⭐⭐⭐⭐⭐ [Fully open-source code and model weights; bridges the long-standing divide between pelvis-centric benchmarks and camera-space deployment.]