Multi-THuMBS: Multi-person Tracking of 3D Human Meshes Beyond Video Shots

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: 3D human mesh recovery, multi-person tracking, multi-shot video, person re-identification, scene geometry prior

TL;DR

Addressing identity drift and trajectory fractures across abrupt camera shot transitions in real-world videos, Multi-THuMBS constructs a shared 3D scene anchor at shot boundaries using a geometric foundation model, linking multi-person trajectories via hybrid 3D-spatial Re-ID and global spatiotemporal smoothing.

Background & Motivation

Tracking and recovering 3D human meshes (HMR) for multiple individuals from unconstrained monocular videos is a fundamental objective in computer vision, underpinning applications from movie post-production and athletic motion capture to embodied intelligence and mixed reality. Recent learning-based global human motion estimation methods have demonstrated remarkable robustness against depth ambiguity, severe occlusion, and physical truncation in single-shot sequences. However, existing multi-person tracking systems operate under the ubiquitous assumption that camera motion is continuous and temporally smooth across all consecutive frames.

In practice, real-world visual content—including broadcast sports, cinematic footage, television sitcoms, and web videos—is edited with frequent, abrupt shot changes. When a shot transition occurs, the camera perspective jumps instantly, inducing massive variations in 2D image scale and visual appearance, changes in the number of visible individuals, and a severe dislocation of the underlying camera coordinate system. Feeding multi-shot sequences directly into existing frameworks triggers fatal tracking failures: models misinterpret viewpoint leaps as drastic physical motions, and their intra-shot temporal smoothness priors inadvertently generate severe artifacts such as erratic joint jitter and foot sliding. Simultaneously, traditional Re-ID algorithms relying exclusively on 2D appearance representations fail to preserve identities across extreme perspective swings.

Prior attempts to overcome discontinuous camera motion have fallen short of a complete solution. Multi-view 3D reconstruction frameworks (e.g., HSfM) can fuse disjoint perspectives into a shared 3D frame, but they strictly require synchronous captures at identical time instances and lack temporal motion modeling. On the other hand, pioneering multi-shot 3D reconstruction methods (e.g., HumanMM, Multishot) are confined to single-person setups, inherently incapable of resolving identity association in multi-person crowded scenes. Core idea: treat the negligible temporal gap between adjacent shot boundary frames as a static multi-view capture, construct a shared 3D scene space via a geometric foundation model (VGGT) as a spatial anchor, and stitch multi-person motions and identities into a unified world coordinate frame through progressive mesh-scene alignment, geometry-driven hybrid Re-ID, and cross-camera reprojection constraints.

Method

Overall Architecture

Multi-THuMBS takes an unconstrained multi-shot monocular video as input, detects transition boundaries, and recovers globally consistent, identity-preserved 3D human mesh (SMPL) trajectories in world coordinates. For two consecutive shots \(S_1\) and \(S_2\) divided at boundary frame \(t_b\) (with adjacent frames \(f_{t_b-1}\) and \(f_{t_b}\)), the framework executes five primary stages: 1. Initial Information Extraction: PySceneDetect segments the video into distinct shots. Within each shot, 4DHumans recovers local SMPL mesh tracklets, Grounded SAM generates individual segmentation masks, and ViTPose extracts 2D keypoints. 2. Boundary Shared 3D Space Construction & Progressive Alignment: VGGT reconstructs dense scene point clouds and camera parameters exclusively for the boundary frames. A three-stage optimization sequentially registers the SMPL meshes into this shared coordinate frame while resolving depth ambiguities. 3. Camera Trajectory & SMPL Parameter Propagation: DROID-SLAM estimates lightweight, smooth camera trajectories independently within each shot. Relative SE(3) transformation matrices computed at the boundary then project all local camera and human mesh trajectories into the unified global space. 4. Geometry-Driven Hybrid Re-Identification: Pairwise 3D Euclidean distances between mesh root locations are fused with UV texture map color differences and joint rotation differences into a joint assignment matrix, solved via the Hungarian algorithm with a spatial distance threshold to reject false matches. 5. Global Spatiotemporal Smoothing & Post-Processing: A joint optimization over all frames enforces temporal joint acceleration smoothness, body shape/pose priors, and cross-camera boundary reprojection consistency, eliminating residual tracking jitters.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Multi-shot Monocular Video"] --> B["Shot Detection & Intra-shot Mesh Recovery<br/>PySceneDetect + 4DHumans"]
    B --> C["Shared 3D Space Construction & Progressive Alignment<br/>VGGT Scene Point Clouds + Three-stage Optimization"]
    C --> D["Camera Pose & SMPL Parameter Propagation<br/>DROID-SLAM + Relative SE3 Transforms"]
    D --> E["Geometry-driven Hybrid Re-Identification<br/>3D Distance + UV Texture + Pose Dissimilarity"]
    E --> F["Cross-camera Reprojection & Global Trajectory Smoothing<br/>Spatiotemporal Joint Optimization"]
    F --> G["Output: Globally Aligned Multi-person 3D Mesh Trajectories"]

Key Designs

1. Boundary Shared 3D Space Construction & Metric Scale Alignment: Converting shot leaps into static multi-view anchors Executing full 3D scene reconstruction across entire video sequences is computationally prohibitive and prone to catastrophic failure in dynamic scenes with moving foregrounds. Multi-THuMBS leverages the physical insight that the temporal interval between boundary frames \(f_{t_b-1}\) and \(f_{t_b}\) is negligible (typically tens of milliseconds), during which human displacement is minimal. Consequently, these two frames are treated as a pseudo-synchronized multi-view capture and processed by VGGT to build a shared 3D point cloud \(P_{t_b-1}, P_{t_b}\) alongside initial camera poses \([R_t \mid C_t]\). To bridge the scale ambiguity between metric SMPL meshes and unscaled neural point clouds, the ratio of maximum centroid-to-vertex distance between SMPL meshes and person-masked point clouds is computed and averaged across all subjects, yielding a unified metric scaling factor applied globally to the scene point cloud.

2. Three-Stage Mesh-to-Scene Progressive Optimization: Decoupling non-convex spatial parameter estimation Directly aligning SMPL meshes \(\{m_{t_b-1}^i, m_{t_b}^i\}\) with scene point clouds \(\{P_{t_b-1}, P_{t_b}\}\) requires optimizing camera poses, human root translations \(\Gamma_t^i\), and global orientations \(\Phi_t^i\). Due to severe parameter entanglement and depth ambiguity, end-to-end joint optimization inevitably falls into sub-optimal local minima. Multi-THuMBS decouples this into a three-stage progressive scheme: - Stage 1 (Translation Initialization): Exploiting the exact pixel-to-point correspondence of VGGT, the 2D pelvis keypoint from ViTPose is mapped directly to its corresponding 3D scene point coordinate, providing a robust initial root translation \(\Gamma_t^i\). - Stage 2 (Rigid Pose Coarse Alignment): Freezing camera parameters, root translation \(\Gamma_t^i\) and orientation \(\Phi_t^i\) are refined over 500 iterations using a 2D reprojection loss \(\mathcal{L}_{\text{2D}}\) against detected keypoints. - Stage 3 (Scene Geometry & Depth Disambiguation): Human parameters and camera extrinsics are jointly optimized over 1500 iterations using a combined objective of 2D reprojection, silhouette boundary distance \(\mathcal{L}_{\text{sil}}\), and 3D depth consistency \(\mathcal{L}_{\text{depth}}\). Specifically, \(\mathcal{L}_{\text{depth}}\) penalizes Euclidean distances between SMPL vertices \(v_k\) and corresponding scene surface points \(p_k\):

\[\mathcal{L}_{\text{depth}} = \sum_{k} \| v_k - p_k \|_2\]

while \(\mathcal{L}_{\text{sil}}\) penalizes vertices projecting outside the segmentation mask via a distance transform field, resolving floating and interpenetration artifacts.

3. Relative SE(3) Dual-Shot Temporal Propagation: Marrying local smoothness with global registration Because feed-forward scene models falter over extended dynamic sequences, intra-shot continuous camera motion is recovered using DROID-SLAM, which delivers lightweight, temporally coherent camera extrinsics \([R^*_t \mid C^*_t]\). To anchor these local trajectories into the global world frame, the relative SE(3) transformation matrix is evaluated at the boundary:

\[\mathcal{R}_t = [\hat{R}_t \mid \hat{C}_t] \cdot [R^*_t \mid C^*_t]^{-1} \quad (t \in \{t_b-1, t_b\})\]

Applying \(\mathcal{R}_{t_b-1}\) to shot \(S_1\) and \(\mathcal{R}_{t_b}\) to shot \(S_2\) seamlessly registers both camera trajectories and human SMPL parameters into the shared global coordinate frame without drift.

4. Geometry-Driven Hybrid Re-Identification: Overcoming visual collapse under extreme perspective shifts Cinematic shot cuts frequently alter viewing angles by \(90^\circ\) or \(180^\circ\), causing purely 2D appearance-based Re-ID descriptors to fail. In contrast, the true 3D spatial position of each individual remains invariant across the infinitesimal boundary interval. Multi-THuMBS computes the pairwise Euclidean distance between root translations in the shared world frame:

\[\mathbb{D}_{ij} = \| {}^w\Gamma_{t_b-1}^i - {}^w\Gamma_{t_b}^j \|_2\]

This spatial metric is fused with appearance dissimilarity \(\mathbb{A}_{ij}\) (pixel-wise color difference over unwrapped 4DHumans UV texture maps) and pose dissimilarity \(\mathbb{P}_{ij}\) (axis-angle differences across mutually visible SMPL joints):

\[\mathbb{U}_{ij} = \lambda_{\text{geo}} \mathbb{D}_{ij} + \lambda_{\text{app}} \mathbb{A}_{ij} + \lambda_{\text{pos}} \mathbb{P}_{ij}\]

Global assignment is solved via the Hungarian algorithm. A spatial gating threshold \(\tau = 1.0\text{ m}\) filters out assignments exceeding 1 meter, gracefully handling subjects entering or exiting the camera frustum across shots.

Loss & Training

The framework operates as a training-free inference-time optimization pipeline using the AdamW optimizer: - Temporal Smoothness & Body Shape Prior Loss: Penalizes 3D joint accelerations across adjacent frames while regularizing SMPL shape parameters \(\beta\) and VPoser latent pose codes \(\zeta\):

\[\mathcal{L}_{\text{smooth}} = \sum_{t} \sum_{i} \| {}^wJ_{t+1}^i - 2\,{}^wJ_t^i + {}^wJ_{t-1}^i \|_2^2 + \lambda_{\beta} \sum_i \| \beta^i \|_2^2 + \lambda_{\text{prior}} \sum_{i,t} \| \zeta_t^i \|_2^2\]

• Cross-Camera Geometric Reprojection Loss: For each matched identity pair \(i \in \mathcal{M}\), human meshes from shot 1 are projected through shot 2's camera and vice versa, enforcing strict multi-view reprojection consistency:

\[\mathcal{L}_{\text{cross}} = \sum_{i \in \mathcal{M}} \left( \| \Pi_f(R_{t_b} J({}^w\phi_{t_b-1}^i) + C_{t_b}) - J_{t_b}^i \|_2 + \| \Pi_f(R_{t_b-1} J({}^w\phi_{t_b}^i) + C_{t_b-1}) - J_{t_b-1}^i \|_2 \right)\]

Balancing these via \(\min \lambda_{\text{smooth}}\mathcal{L}_{\text{smooth}} + \lambda_{\text{cross}}\mathcal{L}_{\text{cross}}\), the entire optimization processes a 150-frame 1080p video in approximately 10 minutes on a single NVIDIA RTX 3090 GPU.

Key Experimental Results

Main Results

Quantitative evaluations are conducted on multi-shot benchmarks synthesized from multi-view multi-person datasets (EgoHumans, EgoBody, Harmony4D), measuring pose accuracy, camera tracking drift (ATE), and cross-shot identity switches (IDs).

Dataset	Method	W-MPJPE↓ (mm)	WA-MPJPE↓ (mm)	MPJPE↓ (mm)	MPVPE↓ (mm)	Accel↓ (\(\text{m/s}^2\))	ATE↓ (m)	IDs↓
EgoHumans	PromptHMR [41]	1778.2	440.9	285.3	364.1	74.3	2.3	10.40
	Multishot [26]	474.1	287.4	347.1	408.2	63.15	-	-
	GVHMR [33]	404.8	204.7	287.4	371.4	59.0	-	-
	HSfM† [23]	544.2	187.7	263.1	294.3	48.7	2.8	3.87
	Ours	279.0	166.0	228.3	262.2	27.3	0.7	0.97
EgoBody	PromptHMR [41]	1228.1	395.3	99.0	133.9	17.4	1.6	1.29
	Multishot [26]	185.1	144.1	147.1	166.5	17.9	-	-
	GVHMR [33]	174.0	133.3	108.0	147.1	14.7	-	-
	HSfM† [23]	113.1	96.3	113.3	123.2	10.9	2.9	0.20
	Ours	99.2	72.8	72.0	94.9	6.0	0.1	0.00
Harmony4D	PromptHMR [41]	1746.3	399.8	675.7	746.0	66.8	2.3	8.00
	Multishot [26]	248.0	231.6	511.2	609.5	37.1	-	-
	GVHMR [33]	244.9	166.7	244.7	334.1	29.6	-	-
	HSfM† [23]	372.0	178.4	225.6	257.6	28.3	3.2	1.58
	Ours	221.0	116.9	215.9	278.3	17.4	0.7	0.46

On unannotated real-world edited videos (AVA, Friends, The Big Bang Theory), motion naturalness is assessed via cross-shot PCK (\(\text{PCK}^*\)), Jitter, and Foot Sliding (FS):

Dataset Setting	Method	\(\text{PCK}^*\)↑ (%)	Jitter↓	Foot Sliding (FS)↓
Edited Videos (AVA / Friends / TBBT)	PromptHMR [41]	62.7	162.44	23.11
	Ours	90.7	31.50	10.70

Ablation Study

Ablation analysis on the EgoHumans benchmark isolating core architectural modules:

Configuration	Description	W-MPJPE↓ (mm)	MPJPE↓ (mm)	MPVPE↓ (mm)	Accel↓ (\(\text{m/s}^2\))	ATE↓ (m)
(1) w/o cam	Removes camera pose optimization	389.7	230.1	264.7	33.7	1.40
(2) w/o post	Removes global spatiotemporal post-processing	311.2	241.6	298.3	47.9	0.77
(3) w/o stage	Disables progressive stages, optimizing all params jointly	491.9	368.1	359.4	33.9	2.75
(4) w/o align	Skips boundary mesh-to-point cloud alignment entirely	882.7	422.4	392.1	34.9	1.40
Full Model (Ours)	Complete proposed pipeline	278.8	228.3	262.1	27.3	0.77

Key Findings

• Boundary Scene Alignment is Foundational: Skipping mesh-to-scene alignment (w/o align) causes trajectory alignment error (W-MPJPE) to skyrocket from 278.8 mm to 882.7 mm (+216%), verifying that absent a shared 3D geometric anchor, isolated shot frames drift into divergent coordinate systems.
• Progressive Staging Prevents Optimization Divergence: Disabling the hierarchical sequence (w/o stage) triggers severe parameter coupling; camera ATE degrades from 0.77 m to 2.75 m and MPJPE spikes by over 140 mm, confirming that translation initialization and rigid pre-alignment are essential before joint depth optimization.
• 3D Spatial Proximity Dominates Re-ID Accuracy: In isolation, our distance-only Re-ID baseline achieves an IDs of 1.66 on EgoHumans, comfortably outperforming dedicated visual Re-ID baselines KPR (2.54) and Pose2ID (4.62). Adding UV appearance and pose dissimilarity further suppresses IDs to 0.97 (and a perfect 0.00 on EgoBody), confirming that geometric proximity provides an invariant anchor against severe visual perspective shifts.

Highlights & Insights

• Boundary Multi-View Formulation as Computational Compression: Rather than applying heavy 3D foundation models across entire video volumes, Multi-THuMBS pinpoints the transition frame pair where time stands still, exploiting instantaneous multi-view geometry with surgical precision and minimal overhead.
• Geometrically Grounded Re-ID: Transcends the traditional limitations of 2D appearance embedding by reducing person matching to spatial proximity in world coordinates, augmented with viewpoint-invariant UV texture and 3D joint rotations.
• Bifurcated Temporal-Global Architecture: Employs DROID-SLAM for smooth, high-frame-rate tracking within continuous shots, bridged effortlessly into global world space via closed-form relative SE(3) transforms at boundaries.

Limitations & Future Work

• Intra-Scene Spatial Overlap Requirement: The system assumes adjacent shots depict overlapping regions of the same physical scene. If an abrupt edit cuts between entirely disjoint scenes (e.g., from an indoor apartment to an outdoor highway), the shared 3D point cloud cannot be formed, rendering geometric tracking invalid.
• Vulnerability to Extreme Background Dynamics: While restricting reconstruction to boundary frames mitigates moving object interference, massive environmental changes (e.g., dynamic lighting strobes, large passing vehicles) can distort VGGT point cloud geometry and degrade registration.
• Cascaded Dependency Pipeline: The framework chains PySceneDetect, 4DHumans, Grounded SAM, ViTPose, VGGT, and DROID-SLAM; undetected shot cuts or missed initial human detections propagate downstream into optimization errors.

Related Work & Insights

• vs. PromptHMR [41] / GVHMR [33]: State-of-the-art single-shot estimators produce catastrophic trajectory breaks and foot sliding across camera cuts; Multi-THuMBS explicitly models shot boundaries, reducing foot sliding artifacts on real edited footage from 23.11 to 10.70.
• vs. HumanMM [47] / Multishot [26]: Prior multi-shot 3D human recovery frameworks assume single-person captures; Multi-THuMBS introduces the first multi-person multi-shot architecture with hybrid 3D Re-ID and cross-camera boundary constraints.
• vs. ShowMak3r [13]: ShowMak3r jointly models humans and neural radiance fields for multi-person TV show reconstruction but targets visual rendering quality; Multi-THuMBS targets metric motion accuracy, physically grounded trajectories, and temporal tracking consistency.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering formulation extending multi-person 3D mesh recovery and geometric Re-ID across abrupt video shot boundaries.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exhaustive evaluation across three multi-person benchmarks and three real edited video domains assessing pose, camera, Re-ID, and motion smoothness.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology presentation, clear progressive optimization formulations, and clean narrative flow.
• Value: ⭐⭐⭐⭐⭐ Highly valuable for automated film production, sports analysis, and world-coordinate multi-human video understanding.