DETRAM: End-to-end DEtection, Tracking and Recovery of HumAn Meshes¶
Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Human Mesh Recovery, Multi-Object Tracking, End-to-end Transformer, Interactive Prompt Tracking, Cross-Frame Query Mechanism
TL;DR¶
DETRAM presents an end-to-end single-stage transformer framework that unifies multi-person detection, 3D SMPL mesh recovery, and temporal tracking within a single decoder pass using persistent detection, tracking, and user-prompt queries.
Background & Motivation¶
In crowded video sequences such as public surveillance, sporting broadcasts, and complex urban environments, recovering temporally consistent 3D human meshes and tracking their identities are vital for mixed reality, sports analytics, and behavior understanding. However, real-world videos exhibit severe challenges: frequent inter-person occlusions, camera movements, boundary truncations, and individuals continuously entering and exiting the view, which frequently cause identity switches and 3D pose degradation.
Historically, multi-person human mesh recovery (HMR) and tracking rely on top-down, multi-stage pipelines: an off-the-shelf detector generates bounding boxes, local person crops are fed into pose regression networks, and a separate heuristic or learned association stage stitches trajectories. This design suffers from cumulative multi-stage errors, high inference latency, and loss of global scene context and inter-person spatial relations due to aggressive image cropping. While recent single-frame full-image DETR-based HMR models alleviate cropping artifacts, they treat frames independently and lack cross-frame identity persistence and promptable user-guided tracking mechanisms.
The entry point of this work is to eliminate the architectural boundary between detection, tracking, and 3D mesh recovery by reformulating them into a joint query set prediction problem inside a single transformer decoder. Core idea: unify detection queries (discovering entering individuals), tracking queries (propagating identity and geometry across frames), and prompt queries (guided by user bounding boxes) within a single feed-forward transformer decoder via Track-Modulated Cross-Attention and a lightweight memory bank.
Method¶
Overall Architecture¶
DETRAM operates as an end-to-end single-stage video processing framework. For each incoming video frame, a DINOv2-based ViT encoder extracts dense spatial visual tokens from full-resolution images (\(1288 \times 1288\)). The unified DETRAM decoder simultaneously receives three complementary query types: detection queries for discovering entering individuals, tracking queries for carrying forward existing identities and poses, and optional prompt queries initialized from user bounding boxes.
Within the decoder, queries first refresh their content embeddings via Track-Modulated Cross-Attention (TMCA) by attending to current detection hypotheses, followed by self-attention across instances and content-conditioned cross-attention over image tokens. Finally, multi-task prediction heads regress 2D bounding boxes, confidence scores, and 3D SMPL parameters (joint rotations \(\theta\), shape parameters \(\beta\), and camera-space translation \(\tau\)). A parameter-free key-value memory bank regularizes temporal identity embeddings across long-term occlusions.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Video Frames<br/>DINOv2 ViT Tokenization"] --> B["Unified Query Anchor Initialization<br/>Detection / Tracking / User Prompts"]
B --> C["Track-Modulated Cross-Attention TMCA<br/>Tracking Queries Aggregate Detection Cues"]
C --> D["DETRAM Unified Decoder<br/>Self-Attention + Conditional Cross-Attention"]
D --> E["Multi-Task Prediction Heads<br/>2D Boxes / 3D SMPL / Confidence"]
E --> F["Lifecycle Management & Memory Bank<br/>Track Promotion / Freeze / Long-Term Retrieval"]Key Designs¶
1. Unified Query Representation and Track-Modulated Cross-Attention (TMCA): Decoupling Discovery from Identity Propagation Conventional multi-person trackers rely on post-hoc matching across frames, which degrades under rapid human deformation. DETRAM parameterizes all three query types with normalized 4D geometric anchor boxes \(A_q = (x_q, y_q, w_q, h_q)\), content embeddings \(C_q\), and sinusoidal positional embeddings \(P_q\). Detection queries employ learnable anchors to identify new humans, tracking queries propagate anchors and embeddings from the previous frame, and prompt queries directly absorb user-supplied bounding boxes.
To adapt tracking queries to sudden camera shifts and non-rigid body motions before main decoding, DETRAM introduces Track-Modulated Cross-Attention (TMCA). Previous tracking anchors \(A_q^{\text{trk}}\) and current detection anchors \(A_k^{\text{det}}\) are projected into a shared embedding space to compute top-\(k\) attention weights: $\(w_{qk} = \mathrm{Softmax}_k\left(\phi(A_q^{\text{trk}})^\top \psi(A_k^{\text{det}})\right), \quad C_q^{\text{trk}} = \sum_{k \in \mathcal{N}_k(q)} w_{qk} C_k^{\text{det}}\)$ This update allows tracking and prompt queries to dynamically absorb compatible appearance and localization cues from current-frame detections, preserving temporal identity continuity without external post-hoc association heuristics.
2. Content-Conditioned Unified Transformer Decoding: Spatially Selective Joint Multi-Person Reasoning Direct full-image regression of 3D meshes risks cross-talk between nearby individuals in crowded scenes. DETRAM concatenates detection and tracking queries to decode them in parallel within the same transformer layers. Queries exchange interpersonal collision and spatial relationship cues in self-attention, and attend to image features using content-conditioned cross-attention: $\(Q_q = \mathrm{Cat}\big(C_q,\, \mathrm{PE}(x_q, y_q) \cdot \mathrm{MLP}^{(\mathrm{csq})}(C_q)\big), \quad K_{x,y} = \mathrm{Cat}\big(F_{x,y},\, \mathrm{PE}(x,y)\big)\)$ The semantic gating projection \(\mathrm{MLP}^{(\mathrm{csq})}\) dynamically modulates positional encodings using query features, focusing cross-attention on accurate human body regions. Multi-branch regressors (confidence \(H_c\), pose \(H_p\), shape \(H_s\), translation \(H_t\), and bounding box \(H_b\)) output full-body 3D SMPL parameters simultaneously, avoiding crop-induced limb truncation.
3. Dynamic Lifecycle Policy and Key-Value Memory Bank: Parameter-Free Long-Term Temporal Stability To handle prolonged occlusions and target reappearance, DETRAM implements a simple non-learned state machine and a key-value memory bank. Detections exceeding confidence \(\tau_{\text{det}}\) with IoU \(< \tau_{\text{iou}}\) against active tracks are instantiated as new tracking queries; when confidence drops, tracks become inactive and their updates freeze; tracks inactive for \(> L_{\text{tol}}\) frames are permanently purged.
For active tracks, past keys \(m_k\) and values \(m_v\) are stored across \(M\) historical slots. Current query keys \(q_k\) compute normalized affinity against memory keys using top-\(k\) softmax: $\(S = \frac{2 m_k^\top q_k - \|m_k\|^2}{\sqrt{D}}, \quad \mathrm{mem}_{b,n} = \sum_m \alpha_{b,n,m} m_v(b,m)\)$ This differentiable mechanism introduces zero additional learnable parameters and consumes only 12–24 KB memory per active track (FP32), providing constant-space temporal regularization invariant to sequence length (\(O(N_t M D)\)).
Loss & Training¶
The overall learning objective unifies detection, mesh reconstruction, and tracking supervision: $\(\mathcal{L} = \lambda_{\text{matching}} \mathcal{L}_{\text{matching}} + \lambda_{\text{dn}} \mathcal{L}_{\text{dn}} + \lambda_{\text{trk}} \mathcal{L}_{\text{trk}}\)$ Detection queries are supervised via Hungarian Matching on 2D boxes, 2D projected joints, and classification confidence scores. To accelerate convergence, a DN-DETR-inspired denoising loss \(\mathcal{L}_{\text{dn}}\) is integrated. Tracking queries are trained by perturbing ground-truth bounding boxes with Gaussian noise and randomly subsampling active tracks to enforce spatial error tolerance and tracking consistency \(\mathcal{L}_{\text{trk}}\).
Training follows a two-stage protocol: initial single-frame training on AGORA, BEDLAM, COCO, PoseTrack, and 3DPW, followed by multi-frame video clip fine-tuning (\(T=4\) or \(8\)) on sequential datasets (BEDLAM, PoseTrack, and 3DPW).
Key Experimental Results¶
Main Results¶
DETRAM was evaluated on PoseTrack21, MuPoTS-3D, BEDLAM, and 3DPW. The table below summarizes multi-person tracking performance on PoseTrack21 and joint tracking and 3D HMR accuracy on BEDLAM:
| Dataset | Method / Baseline | MOTA (↑) | IDF1 (↑) | IDs (↓) | MPJPE (mm, ↓) | PA-MPJPE (mm, ↓) | Inference Latency / FPS |
|---|---|---|---|---|---|---|---|
| PoseTrack21 | 4DHumans (ICCV 2023) | 56.7 | 70.9 | – | – | – | 0.51 FPS |
| PoseTrack21 | CoMotion (ICLR 2025) | 71.4 | 79.5 | – | – | – | 5.68 FPS |
| PoseTrack21 | DETRAM (Ours) | 73.6 | 80.6 | – | – | – | 10.87 FPS |
| PoseTrack21 | DETRAM + Prompts (Ours) | 75.7 | 81.8 | – | – | – | 10.87 FPS |
| BEDLAM | SMPLer-X + ByteTrack | 78.83 | 81.09 | 1660 | 119.19 | 52.62 | 0.463 s/f |
| BEDLAM | MaQ (CVPR 2025) | 95.15 | 97.01 | 128 | 79.88 | 45.56 | 0.027 s/f |
| BEDLAM | CoMotion (ICLR 2025) | 95.22 | 97.39 | 885 | 55.40 | 24.50 | 0.239 s/f |
| BEDLAM | DETRAM (Ours) | 99.18 | 99.37 | 37 | 48.68 | 31.24 | 0.069 s/f |
On 3DPW benchmarks, DETRAM attains 99.8 MOTA, 99.9 IDF1, 79.1 HOTA, and 0 ID switches on Dyna3DPW tracking, alongside 60.0 mm MPJPE and 70.1 mm PVE for 3D mesh reconstruction, outperforming previous temporal tracking and HMR baselines.
Ablation Study¶
The ablation below evaluates the contributions of persistent tracking queries, the memory bank, TMCA, and content-conditioned spatial modulation:
| Configuration / Variant | PoseTrack21 MOTA (↑) | PoseTrack21 IDF1 (↑) | BEDLAM MOTA (↑) | BEDLAM IDs (↓) | BEDLAM MPJPE (mm, ↓) | Note |
|---|---|---|---|---|---|---|
| DETRAM (Full Model) | 73.6 | 80.6 | 99.18 | 37 | 48.68 | Full model with Track-Q, Mem Bank, and TMCA |
| w/o Track-Q (Detection-only) | 68.3 | 77.1 | 96.70 | 58 | 46.91 | Without tracking queries, MOTA drops sharply by 5.3 |
| w/ Track-Q, w/o Mem Bank | 73.3 | 80.3 | 99.10 | 32 | 47.33 | Tracking queries provide primary temporal stability |
| TMCA \(\to\) Direct Propagation | 71.7 | 79.6 | 99.16 | 47 | 49.37 | Bypassing TMCA degrades tracking consistency |
| \(\mathrm{MLP}^{(\mathrm{csq})} \to\) Identity | 70.2 | 77.1 | 98.38 | 305 | 52.73 | Without spatial modulation, ID switches jump to 305 |
Key Findings¶
- Complementary Roles of Tracking Queries and Memory Bank: Persistent tracking queries deliver the bulk of identity consistency (reaching 73.3 MOTA on PoseTrack21). The memory bank further recovers occluded targets (boosting MOTA to 73.6). Although recalling challenging occluded instances slightly shifts average reconstruction error (MPJPE from 47.33 mm to 48.68 mm), trajectory completeness and overall MOTA improve significantly.
- Superior Robustness in Crowded Scenarios: In stratified evaluations on PoseTrack21 sequences with \(>21\) persons, DETRAM achieves a MOTA of 66.83 versus CoMotion's 60.53 (over 10% absolute gain), proving the resilience of full-frame transformer cross-attention over cascaded box cropping.
- User Prompting Interactivity: When user bounding box prompts are provided upon false negatives, prompt queries effectively propagate through subsequent frames, pushing PoseTrack21 MOTA from 73.6 to 75.7 and proving seamless human-in-the-loop controllability.
Highlights & Insights¶
- Unified Single-Stage Transformer Architecture: Eliminates multi-stage "detector \(\to\) crop \(\to\) mesh regression \(\to\) ReID tracking" pipelines, achieving 10.87 FPS (over \(21\times\) faster than 4DHumans) in a single decoder forward pass.
- Multi-Role Persistent Query Paradigm: Seamlessly unifies detection queries, tracking queries, and prompt queries under normalized 4D geometric anchors, allowing the same model to support both automated tracking and user-directed interactive video analysis.
- Constant-Space Temporal Regularization: The parameter-free key-value memory bank requires only 12–24 KB memory per active track, providing long-term temporal consistency without growing with video sequence length (\(O(N_t M D)\)).
Limitations & Future Work¶
- Lack of Explicit Temporal Smoothness Constraints: The model conducts framewise inference; despite consistent tracking IDs, high-frequency camera shakes can induce minor frame-to-frame joint jitter. Incorporating motion kinematics priors or temporal velocity loss could further improve physical plausibility.
- Limited Prompt Modalities: Current prompt queries only support 2D bounding boxes. Extending prompt interfaces to point clicks, natural language instructions, or trajectory sketches would expand interactive utility in dense scenes.
- Reliance on Contiguous Video Annotations: Training requires continuous ground-truth tracks, restricting large-scale pre-training on raw unannotated internet videos.
Related Work & Insights¶
- vs CoMotion (ICLR 2025): CoMotion relies on separate per-frame detectors and pose update networks; DETRAM conducts joint detection, tracking, and HMR within a single decoder, cutting latency in half and uniquely enabling promptable tracking.
- vs MaQ (CVPR 2025): MaQ uses clip-level motion queries for offline whole-body batch reconstruction; DETRAM processes streaming video framewise with online prompt re-initialization.
- vs SAM 3 (2025): SAM 3 targets promptable 2D mask tracking; DETRAM unifies promptable tracking with full 3D parametric body mesh (SMPL) recovery in video.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant unification of detection, tracking, and prompt queries inside a single-stage transformer decoder.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation on PoseTrack21, MuPoTS-3D, BEDLAM, and 3DPW with detailed ablations and crowd stratification.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, clear figures, and explicit formulation of architectural trade-offs.
- Value: ⭐⭐⭐⭐⭐ Offers an efficient, highly practical foundation for real-time video motion capture, sports analytics, and interactive metaverse systems.