Skip to content

Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration

Conference: ECCV 2026
Paper: ECCV Official
Area: Autonomous Driving / Human Understanding
Keywords: Cross-Modal Motion Capture, LiDAR-Camera Fusion, Calibration-Free, Bottleneck Attention, Noise-Resilient Trajectory Tracking

TL;DR

Sen-Cap presents a calibration-free 3D human motion capture framework that supports plug-and-play arbitrary numbers of LiDARs and cameras while remaining resilient against heavy point cloud clutter, estimating local poses via human-centric feature alignment with bottleneck attention and tracking global translation via iterative offset shrinkage.

Background & Motivation

Accurate and robust 3D human motion capture plays a foundational role in embodied AI, autonomous driving interactions, augmented/virtual reality, and sports analytics. Conventional markerless systems largely rely on single-modality sensors: monocular or multi-view cameras lack metric depth perception and suffer from scale ambiguity under severe occlusions, whereas LiDAR provides reliable metric depth and geometric boundaries but lacks dense surface texture and suffers from acute point sparsity at long ranges. Fusing camera appearance cues with LiDAR geometric structure is thus a natural path toward robust human motion estimation.

However, existing multi-modal pipelines face severe bottlenecks when deployed in unconstrained real-world environments. First, conventional methods depend strictly on explicit inter-sensor extrinsic calibration, requiring static configurations with highly overlapping viewpoints; any perspective change or dynamic layout disturbance triggers catastrophic error propagation. Even recent calibration-free approaches like FreeCap rely on on-the-fly optimization over matched human keypoints to solve a coarse extrinsic matrix, which is computationally heavy, easily fails under sparse human points, and strictly limits setup to a single LiDAR. Second, real-world point clouds are contaminated by severe clutter, ground reflections, and object noise. Existing methods typically center point clouds to their physical centroid and predict a single offset to the human pelvis; under heavy noise or occlusion, the centroid shifts drastically away from the human root, forcing the network to search across an abnormally large candidate space and producing severe systematic trajectory drift.

This paper tackles these challenges by recognizing that human kinematic geometry is intrinsically invariant to external sensor viewpoints, allowing heterogeneous observations to be mapped directly into a unified human-centric reference frame without extrinsic calibration; furthermore, global trajectory drift can be eliminated through progressive spatial contraction. Core idea: construct a calibration-free multi-modal framework, Sen-Cap, that recovers local pose and shape across arbitrary numbers of uncalibrated cameras and LiDARs via human-centric space alignment and bottleneck attention, while stabilizing global translation under extreme clutter through a noise-resilient iterative refinement tracker.

Method

Overall Architecture

Sen-Cap takes as input uncalibrated point cloud sequences \(\{x_{pc}^{L_i}\}_{i=1}^{N_L}\) from an arbitrary number of movable LiDARs and normalized 2D joint sequences \(\{x_{2d}^{C_j}\}_{j=1}^{N_C}\) from movable cameras. It directly outputs full SMPL human mesh parameters, including shape \(\hat{\beta} \in \mathbb{R}^{10}\), local pose \(\hat{\theta} \in \mathbb{R}^{23 \times 3}\), global orientation \(\hat{\tau} \in \mathbb{R}^3\), and metric global translation \(\hat{\Gamma} \in \mathbb{R}^3\).

The architecture is decoupled into two coordinated branches: 1. Unified Across-Sensor Motion Estimator (UAME): Reconstructs human local pose \(\hat{\theta}\) and shape \(\hat{\beta}\) by aligning multi-modal sensor tokens into a canonical human-centric space and aggregating reliable cues via compact bottleneck cross-attention without requiring inter-sensor calibration; 2. Noise-Resilient Trajectory Tracker (NTT): Solves point cloud centroid shift caused by background clutter through multi-step iterative residual offset prediction guided by estimated 3D joints, progressively contracting the displacement search space to estimate global translation \(\hat{\Gamma}\).

The overall workflow is illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Multi-Modal Inputs<br/>Uncalibrated Point Clouds + 2D Keypoints"] --> B["Human-Centric Space Alignment<br/>GRU Predicts Rotation to Canonical Frame"]
    B --> C["Adaptive Sensor Fusion<br/>Learnable Latent Token Bottleneck Cross-Attention"]
    C --> D["Sensor Dropout Strategy<br/>Random Token Masking for Robustness"]
    D --> E["Local Pose & Shape Regression<br/>Predicts SMPL Parameters & Local 3D Joints"]
    E --> F["Noise-Resilient Trajectory Tracker<br/>Iterative Residual Offset Shrinkage"]
    F --> G["Final Predictions<br/>Global 3D Human Mesh & Metric Trajectory"]

Key Designs

1. Human-Centric Space Alignment: Eliminating Inter-Sensor Calibration Dependency

Standard multi-sensor fusion pipelines project observations into a shared coordinate system using rigid extrinsic matrices, which easily degrades when sensors move or vibrate. Sen-Cap exploits the kinematic invariance of the human body by mapping all sensor inputs into an anatomy-aligned human-centric frame \(\mathcal{H}\). For each LiDAR \(L_i\), 256 sampled points are processed by a PointNet encoder into 3D joint representations \(x_{3d}^{L_i}\), while a Gated Recurrent Unit (GRU) predicts the sensor-to-human global rotation vector \(\hat{\tau}_i\). The 3D representations are transformed into the canonical human-centric coordinate system via inverse rotation:

\[x_{3d}^{H_i} = R(\hat{\tau}_i)^{-1} x_{3d}^{L_i}\]

where \(R(\cdot)\) converts the rotation vector into a \(3 \times 3\) rotation matrix. Multi-view alignment is supervised with loss \(\mathcal{L}_{align}\) across both sensor and human-centric coordinate systems, enforcing consistency on 3D joint coordinates alongside continuous 6D global rotation representations. Concurrently, 2D keypoints \(x_{2d}^{C_j}\) normalized by bounding box center and scale are encoded into the shared feature space. This uncouples multi-sensor processing from explicit physical calibration and allows sensors at arbitrary viewpoints to communicate in a unified anatomical space.

2. Adaptive Sensor Fusion: Bottleneck Attention for Flexible Sensor Scalability

In real-world setups, sensor reliability fluctuates unpredictably due to lighting shifts, dynamic occlusions, or sensor packet loss. Unrestricted pairwise cross-attention incurs quadratic computational complexity with respect to the number of sensors and easily propagates noise from corrupted streams. Sen-Cap introduces an adaptive fusion module based on a learnable latent bottleneck token.

Temporal motion features \(F_{3d}\) from LiDAR and visual motion features \(F_{2d}\) from cameras are extracted by respective GRUs and concatenated into \(F_m = [F_{3d}, F_{2d}]\). Rather than performing unconstrained cross-modal interactions, a learnable latent token \(F_{token} \in \mathbb{R}^{1 \times D}\) acts as the Query, while concatenated multi-modal features \(F_m\) serve as Keys and Values:

\[F_{fusion} = \text{CrossAttn}(Q = F_{token}, K = F_m, V = F_m)\]

This information bottleneck forces multi-sensor cues to be compressed into a compact latent representation. The attention mechanism dynamically assigns higher weights to informative, uncorrupted viewpoints while filtering out degraded signals. The fused token \(F_{fusion}\) is then decoded by an output GRU to predict SMPL parameters \(\hat{\beta}\) and \(\hat{\theta}\). Crucially, because \(F_{token}\) has a fixed channel size regardless of how many sensors are connected, the model supports seamless plug-and-play scaling of cameras and LiDARs without weight alteration.

3. Sensor Dropout Strategy: Generalization under Sensor Dropouts and Failures

To ensure robust performance when sensors drop out, become occluded, or operate in sparse configurations, Sen-Cap incorporates a Sensor Dropout regularizer during training. For each training sample, feature tokens across LiDAR and camera streams are randomly masked with a preset probability prior to bottleneck fusion. This simulates diverse failure modes ranging from single-sensor degradation to total modality absence. The training constraint forces the bottleneck token to distill redundant structural priors from incomplete signals, allowing a unified model trained on 3C+3L to generalize directly to 1C+1L, LiDAR-only, or camera-only setups without fine-tuning.

4. Noise-Resilient Trajectory Tracker: Iterative Refinement for Robust Global Localization

Recovering global translation \(\hat{\Gamma}\) is challenging in cluttered outdoor scenes. Prior works normalize the point cloud to its geometric centroid \(x_{pc}\) and regress a single offset to the human root joint. Under severe noise, ground reflections, or surrounding clutter, \(x_{pc}\) shifts drastically away from the human pelvis, forcing the network to explore an abnormally large displacement space, resulting in systematic under-estimation and drift.

The Noise-Resilient Trajectory Tracker (NTT) introduces an iterative contraction mechanism. Initializing the reference center at \(x_{pc}^0 = x_{pc}\) alongside estimated local 3D joints \(\hat{x}_{3d}^L\), offset predictor \(E_O\) progressively computes residual updates:

\[\Delta \hat{\Gamma}^n = E_O(x_{pc}^{n-1}, \hat{x}_{3d}^L), \quad x_{pc}^n = x_{pc}^{n-1} + \Delta \hat{\Gamma}^n\]

After \(N\) iterative steps, the final global translation is obtained as:

\[\hat{\Gamma} = x_{pc} + \sum_{n=1}^N \Delta \hat{\Gamma}^n\]

Each step nudges the coordinate origin closer to the true human pelvis, shrinking the required offset magnitude and search variance, which prevents peripheral clutter points from pulling the root estimate off-target.

Loss & Training

The entire network is trained end-to-end with a composite objective combining alignment loss \(\mathcal{L}_{align}\), pose supervision \(\mathcal{L}_{pose}\), and trajectory tracking loss \(\mathcal{L}_{tr}\):

\[\mathcal{L} = \mathcal{L}_{align} + \mathcal{L}_{pose} + \mathcal{L}_{tr}\]

The pose loss enforces L1 distance on shape and pose parameters: \(\mathcal{L}_{pose} = \|\beta - \hat{\beta}\|_1 + \|\theta - \hat{\theta}\|_1\). The trajectory loss combines absolute translation error with a step penalty \(\mathcal{L}_{dtr}\) that suppresses excessively large incremental steps:

\[\mathcal{L}_{tr}(\Gamma) = \|\hat{\Gamma} - \Gamma\|_1 + \lambda \frac{1}{N} \sum_{n=1}^N \mathcal{L}_{dtr}(\Delta \hat{\Gamma}^n)\]

The step loss penalizes individual displacement magnitudes exceeding threshold \(\delta\), guaranteeing temporal trajectory smoothness and numerical stability under heavy clutter.

Key Experimental Results

Main Results

Sen-Cap is evaluated on the multi-sensor FreeMotion benchmark and the large-scale multi-person Human-M3 benchmark under both standard Training View (T) and unseen Novel Camera View (N) protocols. Evaluated baselines include vision-based methods (WHAM, GVHMR, GENMO, PromptHMR), LiDAR-based methods (LiveHPS, LiveHPS++), and the hybrid calibration-free baseline FreeCap. Metrics include joint/vertex error J/V Err(PS), translation-included global joint/vertex error J/V Err(PST), angular rotation error Ang Err (deg), acceleration error Accel Err (\(\text{m/s}^2\)), and unidirectional Chamfer distance SUCD (mm).

Dataset Protocol (View) Method Modality J/V Err(PS) ↓ (mm) J/V Err(PST) ↓ (mm) Ang Err ↓ (°) Accel Err ↓ (\(\text{m/s}^2\)) SUCD ↓ (mm)
FreeMotion Training View (T) WHAM (CVPR'24) Camera (C) 82.83 / 97.55 - 12.24 4.51 -
FreeMotion Training View (T) LiveHPS (CVPR'24) LiDAR (L) 59.30 / 73.12 100.81 / 109.09 13.10 6.18 4.97
FreeMotion Training View (T) LiveHPS++ (ECCV'24) LiDAR (L) 54.15 / 57.74 88.91 / 97.53 11.85 4.37 3.47
FreeMotion Training View (T) FreeCap (AAAI'25) LiDAR+Camera (L+C) 53.31 / 65.50 95.97 / 102.91 11.14 5.97 4.82
FreeMotion Training View (T) Ours (Single-LiDAR) LiDAR (L) 52.98 / 56.41 84.37 / 91.62 11.21 4.28 3.21
FreeMotion Training View (T) Ours (Full Model) LiDAR+Camera (L+C) 47.46 / 57.84 75.16 / 81.65 9.94 3.01 3.83
FreeMotion Novel View (N) FreeCap (AAAI'25) LiDAR+Camera (L+C) 57.37 / 69.61 99.20 / 106.15 11.74 6.08 4.96
FreeMotion Novel View (N) Ours (Full Model) LiDAR+Camera (L+C) 52.25 / 63.05 79.05 / 85.82 10.80 3.16 3.93
Human-M3 Training View (T) FreeCap (AAAI'25) LiDAR+Camera (L+C) 55.45 / 68.52 96.47 / 102.67 9.14 9.60 6.66
Human-M3 Training View (T) Ours (Full Model) LiDAR+Camera (L+C) 42.59 / 52.09 73.50 / 77.53 5.71 10.18 5.74
Human-M3 Novel View (N) FreeCap (AAAI'25) LiDAR+Camera (L+C) 56.24 / 69.25 96.45 / 102.46 9.17 9.59 6.58
Human-M3 Novel View (N) Ours (Full Model) LiDAR+Camera (L+C) 42.64 / 52.28 73.41 / 77.51 5.76 10.15 5.74

Ablation Study

Ablations on FreeMotion dissect the impact of the feature alignment space, fusion mechanism in UAME, and refinement iterations in NTT:

Component Category Configuration Scheme J/V Err(PS) ↓ (mm) J/V Err(PST) ↓ (mm) Ang Err ↓ (°) Accel Err ↓ (\(\text{m/s}^2\)) SUCD ↓ (mm)
Feature Space (UAME) LiDAR-Centric Space 68.73 / 81.42 92.38 / 100.81 12.75 3.39 5.11
Feature Space (UAME) Human-Centric Space (Ours) 52.25 / 63.05 78.87 / 85.66 10.80 3.12 3.91
Feature Fusion (UAME) Learnable Linear Fusion 60.63 / 72.41 85.49 / 93.09 11.73 3.27 4.45
Feature Fusion (UAME) Fixed-Weight Summation 61.37 / 73.13 85.97 / 93.55 11.70 3.28 4.53
Feature Fusion (UAME) Bottleneck Attention (Ours) 52.25 / 63.05 78.87 / 85.66 10.80 3.12 3.91
Trajectory Refinement (NTT) 1 Iteration (\(\text{repeat} \times 1\)) 52.25 / 63.05 87.88 / 94.01 10.80 5.05 4.26
Trajectory Refinement (NTT) 2 Iterations (\(\text{repeat} \times 2\)) 52.25 / 63.05 82.42 / 88.78 10.80 4.18 3.54
Trajectory Refinement (NTT) 3 Iterations (\(\text{repeat} \times 3\), Default) 52.25 / 63.05 78.87 / 85.66 10.80 3.12 3.91
Trajectory Refinement (NTT) 4 Iterations (\(\text{repeat} \times 4\)) 52.25 / 63.05 74.88 / 82.20 10.80 3.17 3.81
Trajectory Refinement (NTT) 5 Iterations (\(\text{repeat} \times 5\)) 52.25 / 63.05 74.63 / 82.00 10.80 3.17 3.75

Key Findings

  • Human-centric alignment is the key enabler for calibration-free tracking: Reverting from human-centric coordinates to a LiDAR-centric space causes J/V Err(PS) to degrade from 52.25/63.05 mm to 68.73/81.42 mm, demonstrating that decoupling sensor placement from pose regression is essential for multi-modal generalization.
  • Bottleneck attention suppresses cross-modal pollution: Simple learnable linear projection produces a 60.63 mm error, while compressing heterogeneous cues into a shared latent query forces reliability-aware reweighting, filtering out noisy modalities.
  • Progressive convergence of iterative refinement: Increasing NTT iterations from 1 to 3 reduces J/V Err(PST) from 87.88 mm to 78.87 mm and acceleration error from 5.05 to 3.12 \(\text{m/s}^2\). Local pose metrics remain perfectly unchanged, validating the orthogonality between local kinematics and global translation.
  • Dramatic resilience against synthetic clutter: In Level 3 noise tests (large ShapeNet objects up to 1.0 m injected into human point clouds), the baseline without NTT collapses to 367.51 mm PST error and 94.62 \(\text{m/s}^2\) acceleration error; enabling NTT (\(N=3\)) restrains error to 158.53 mm and 6.71 \(\text{m/s}^2\).

Highlights & Insights

  • Framing calibration-free alignment as kinematic canonicalization: Rather than calculating high-dimensional inter-sensor transformation matrices through numerical optimization, Sen-Cap predicts human body orientation in each sensor coordinate and rotates features into a canonical human-centric frame, bypassing extrinsic calibration entirely.
  • Bottleneck latent token as an elastic sensor interface: Querying multi-modal features with a fixed-dimension learnable token decouples input sensor count from downstream regression layers, enabling dynamic plug-and-play scalability across arbitrary sensor setups.
  • Iterative contraction resolves centroid bias: Severe clutter pulls point cloud centroids away from human pelvises; iterative residual shrinkage continuously contracts the search radius, effectively preventing large-offset under-estimation and trajectory drift.

Limitations & Future Work

  • Reliance on upstream 2D pose estimators: Although unconstrained by camera extrinsics, the camera branch requires reliable 2D keypoints from upstream detectors (e.g., ViTPose). Performance degrades if optical sensors suffer severe underexposure or blinding glare.
  • Multi-person identity association in crowded scenes: The framework relies on a memory bank matching module to link identities across viewpoints. Cross-view matching under prolonged heavy occlusions and dense crowd crossings remains challenging.
  • vs FreeCap (AAAI 2025): FreeCap estimates a coarse extrinsic matrix on the fly via matched keypoints, which suffers from heavy computational overhead, fails under sparse point clouds, and only supports a single LiDAR. Sen-Cap completely abandons extrinsic calibration via human-centric canonicalization, reducing novel-view joint error on FreeMotion from 57.37 mm to 52.25 mm while supporting multiple LiDARs.
  • vs LiveHPS / LiveHPS++ (CVPR 2024 / ECCV 2024): LiveHPS relies on single-step centroid offset prediction for global translation, which drifts severely when non-human geometric noise shifts the centroid. Sen-Cap's NTT mitigates this failure, slashing Level 3 clutter translation error from 367.51 mm down to 158.53 mm.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering calibration-free multi-modal framework supporting arbitrary LiDARs and cameras via human-centric alignment and bottleneck attention]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated on FreeMotion, Human-M3, plus zero-shot cross-domain tests on LiDARHuman26M and RELI11D, with detailed noise and scalability ablations]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, elegant architecture, transparent motivation, and well-designed ablation studies]
  • Value: ⭐⭐⭐⭐⭐ [Delivers a robust, plug-and-play perception solution for roadside perception, autonomous driving, and multi-robot collaboration]