SynHMR: Synergistic Joint-Mesh Modeling for LiDAR-based Human Mesh Reconstruction¶

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/ShiRui1208/SynHMR
Area: Autonomous Driving
Keywords: 3D Human Mesh Reconstruction, LiDAR Point Cloud, Synergistic Joint-Mesh Modeling, Transformer, Topology-Aware GCN

TL;DR¶

Addressing the severe error propagation inherent to decoupled pipelines that estimate skeletal joints before reconstructing meshes under sparse LiDAR point clouds, SynHMR proposes a synergistic joint-mesh modeling framework using a Geometry-Aware Transformer with noise-augmented learning to enable bidirectional mutual refinement between skeleton and surface, achieving state-of-the-art accuracy across three benchmarks.

Background & Motivation¶

LiDAR-based 3D Human Mesh Reconstruction (HMR) plays a foundational role in autonomous driving environment perception, robotic interaction, and complex scene behavioral analysis. Compared to traditional RGB and RGB-D sensors that are highly susceptible to drastic illumination variations, severe weather, limited detection range, and privacy violations, LiDAR provides robust, long-range, and privacy-preserving 3D spatial geometry observations. However, in real-world autonomous driving scenarios, the human point clouds captured at medium-to-long distances are inherently sparse—often containing only tens to hundreds of unorganized points. Particularly on slender anatomical structures such as limbs, point clouds become severely incomplete and noisy, imposing severe challenges for recovering high-fidelity human body surfaces and accurate poses.

Most existing LiDAR-based HMR approaches adhere to a conventional decoupled, stepwise pipeline: they first regress 3D skeletal joint positions from point cloud features and subsequently reconstruct the surface mesh (e.g., SMPL model parameters or explicit vertex sets) conditioned on the estimated pose. Under sparse and incomplete point clouds, this one-way causal pipeline exhibits a catastrophic vulnerability: estimating joints upfront fails to exploit local surface geometry and boundary cues to correct early pose inaccuracies. Consequently, the downstream mesh reconstruction stage inevitably inherits, propagates, and amplifies initial joint errors. Even when coarse-to-fine progressive strategies are employed, they continue to treat joint estimation and mesh generation as separate steps, overlooking the intrinsic spatial and anatomical constraints that the human surface mesh can impose on the internal skeleton.

The essential breakthrough lies in rethinking and unifying the symbiotic relationship between the skeleton and the surface mesh. Skeletal joints provide a strong structural topological prior that ensures biomechanical plausibility, while surface mesh vertices directly anchor the local geometric observations of the point cloud, offering reliable spatial envelopes to pull deviated joints back into anatomically correct positions. Core idea: formulate both skeletal joints and surface sampled vertices as position-aware queries within a unified representation space, iteratively refining them through a Geometry-Aware Transformer with bidirectional self-attention and noise-augmented learning to eliminate unidirectional error cascading.

Method¶

Overall Architecture¶

SynHMR accepts a single-frame sparse LiDAR point cloud as input and directly reconstructs the parametric SMPL mesh and 3D joint locations end-to-end. The overall system operates across three synergistic processing stages: 1. Synergistic Query Initialization & Noise-Augmented Learning: Farthest Point Sampling (FPS) selects representative surface vertices on the canonical T-pose SMPL template. PointNet++ extracts multi-level geometric features and a global semantic feature from the input point cloud. A dual-branch MLP decoder initializes coarse coordinates for 24 skeletal joints and $N_v$ sampled vertices. During training, Gaussian perturbations are injected into the joints to construct a noise-augmented hybrid query set; 2. Geometry-Aware Transformer Iterative Refinement: Joint, vertex, and perturbed queries are mapped to content embeddings and augmented with Gaussian Fourier Positional Encoding (GFPE). Cascaded decoder layers apply self-attention to facilitate mutual cross-talk between skeleton and surface, followed by cross-attention over multi-level point cloud features to anchor refinement in actual 3D observations, updating coordinates via per-layer offset MLPs; 3. Topology-Aware Mesh Resolver: Refined joint coordinates, geometry-aware joint features, and repeated global features are concatenated and fed into a Spatial Graph Convolutional Network (GCN) structured according to human skeletal anatomy, directly regressing 6D continuous joint rotations for final reconstruction via a differentiable SMPL layer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Sparse LiDAR Point Cloud<br/>N=512 Normalized Input"] --> B["Synergistic Query Initialization & NAL<br/>PointNet++ Backbone / FPS Sampling / Perturbed Joints"]
    B --> C["Geometry-Aware Transformer<br/>GFPE Positional Encoding / Joint-Mesh Self-Attention / Multi-Level Cross-Attention"]
    C --> D["Topology-Aware Mesh Resolver<br/>Multi-Scale Fusion / Spatial GCN Skeletal Graph / Differentiable SMPL"]
    D --> E["Dense 3D Human Recovery<br/>SMPL Mesh V_Mesh (6890) and Skeleton Joints J_SMPL (24)"]

Key Designs¶

1. Synergistic Query Initialization & Noise-Augmented Learning: Breaking the Stepwise Decoupled Bottleneck

Decoupled pipelines treat joint prediction as an isolated prerequisite for mesh reconstruction, causing early pose estimation errors to irrevocably contaminate surface geometry. To achieve symmetric co-optimization, SynHMR uses Farthest Point Sampling (FPS) on the neutral T-pose SMPL template to pick $N_v = 256$ representative surface vertices $I_{sample}$, ensuring uniform spatial coverage across the torso and limbs. Given the normalized point cloud $P \in \mathbb{R}^{512 \times 3}$, a PointNet++ encoder extracts intermediate multi-level point features $F_{pc}$ from three hierarchical Set Abstraction (SA) layers and a 1024-dimensional global semantic feature $F_g$. A dual-head MLP decoder directly estimates coarse coordinates for 24 skeletal joints $J \in \mathbb{R}^{24 \times 3}$ and the $N_v$ sampled vertices $V \in \mathbb{R}^{N_v \times 3}$ from $F_g$.

To prevent the network from developing an over-reliance on initial joint guesses and to stimulate surface-to-skeleton corrective cues, a Noise-Augmented Learning (NAL) strategy is introduced during training. Gaussian perturbations are added to the initial joints: $$\hat{J} = J + \epsilon, \quad \epsilon \sim \mathcal{N}(0, \sigma^2)$$ Learnable content embeddings $E_J \in \mathbb{R}^{24 \times D}$, $E_{\hat{J}} \in \mathbb{R}^{24 \times D}$, and $E_V \in \mathbb{R}^{N_v \times D}$ (with hidden dimension $D = 256$) are assigned to clean joints, noisy joints, and surface vertices, respectively, forming a combined query set of size $N_q = N_v + 48$. This setup forces the network to learn robust joint-mesh spatial dependencies and use surface vertices to actively correct corrupted joint estimates.

2. Geometry-Aware Transformer: High-Frequency Position Encoding and Bidirectional Mutual Refinement

Under extreme point sparsity, raw 3D coordinates lack high-frequency discriminative power in self-attention layers. SynHMR stacks $L = 6$ Geometry-Aware Transformer decoder layers, equipped with a Gaussian Fourier Positional Encoding (GFPE) module. Using a random Gaussian projection matrix $B \in \mathbb{R}^{3 \times (D/2)}$, coordinates $x$ are mapped to a high-dimensional Fourier space and processed by an MLP: $$\gamma(x) = [\sin(2\pi x B), \cos(2\pi x B)] \in \mathbb{R}^D, \quad \text{GFPE}(x) = \text{MLP}(\gamma(x)) \in \mathbb{R}^D$$ GFPE is applied to both query positions and the coordinates of multi-level point features $F_{pc}$, injecting fine-grained spatial sensitivity.

Inside each decoder layer, self-attention models interactions across joint, perturbed joint, and vertex queries. The skeleton transfers kinematic topological structure to guide vertex queries toward anatomically plausible configurations, while vertex queries leverage surface contact to constrain joint queries, pulling outlying joints back within the body envelope. Next, cross-attention queries the projected multi-level point features $F_{pc} = [W_1 F^{(1)}, W_2 F^{(2)}, W_3 F^{(3)}] \in \mathbb{R}^{N_s \times D}$, grounding hybrid queries in raw LiDAR observations. Finally, an offset MLP predicts 3D coordinate displacements $\Delta E_{pos}^{(l)}$, updating positions iteratively: $$E_{pos}^{(l)} = E_{pos}^{(l-1)} + \Delta E_{pos}^{(l)}$$ After $L$ layers, the noisy queries are discarded, yielding refined joints $J^* \in \mathbb{R}^{24 \times 3}$, refined vertices $V^* \in \mathbb{R}^{N_v \times 3}$, and refined joint features $E_J^* \in \mathbb{R}^{24 \times D}$.

3. Topology-Aware Mesh Resolver: Skeletal Graph-Driven Articulated Regression

Rather than mapping unconstrained joint features to body rotations using flat perceptrons, SynHMR constructs an anatomical graph convolutional resolver. For each of the 24 joints, multi-scale features are concatenated along the feature dimension: $$F_{GCN} = \text{Concat}\big(J^*,\, E_J^*,\, \text{Repeat}(F_g)\big) \in \mathbb{R}^{24 \times (3 + D + 1024)}$$ This fuses explicit 3D joint coordinates, geometry-aware local surface context, and global body semantics into a unified joint representation.

A skeletal graph matching human anatomical connections propagates information across neighboring joints via Spatial GCN layers. This structural constraint enforces biological plausibility and prevents unnatural joint twists. The GCN outputs a continuous 6D rotation representation for each joint, converted to rotation matrices $\theta^* \in \mathbb{R}^{24 \times 3 \times 3}$. Finally, $\theta^*$ is passed to the differentiable SMPL model using Linear Blend Skinning (LBS) to produce the dense 3D mesh $V_{Mesh} \in \mathbb{R}^{6890 \times 3}$ and the final joints $J_{SMPL} \in \mathbb{R}^{24 \times 3}$.

Loss & Training¶

The network is trained end-to-end on an NVIDIA RTX 3090 Ti GPU using the Adam optimizer with a batch size of 16 for 200 epochs. The initial learning rate is $1 \times 10^{-4}$, weight decay is $1 \times 10^{-4}$, and the GCN dropout rate is set to 0.5. The total loss objective is defined as: $$\mathcal{L}_{total} = \lambda_{vertex} \mathcal{L}_{vertex} + \lambda_J \mathcal{L}_J + \lambda_{J_{SMPL}} \mathcal{L}_{J_{SMPL}} + \lambda_\theta \mathcal{L}_\theta$$ The loss terms are formulated as follows: 1. Refined Vertex Loss: $\mathcal{L}_{vertex} = \frac{1}{N_v} \sum_{k=1}^{N_v} \|V_k^* - V_{GT,k}\|_2^2$, directly supervising refined sampled vertices to ensure geometric surface alignment; 2. Refined Joint Loss: $\mathcal{L}_J = \frac{1}{24} \sum_{j=1}^{24} \|J_j^* - J_{GT,j}\|_2^2$, providing direct coordinate supervision on Transformer joint queries; 3. SMPL Joint Loss: $\mathcal{L}_{J_{SMPL}} = \frac{1}{24} \sum_{j=1}^{24} \|J_{SMPL,j} - J_{GT,j}\|_2^2$, supervising the final joints produced by the SMPL model; 4. Pose Rotation Loss: $\mathcal{L}_\theta = \frac{1}{24} \sum_{j=1}^{24} \|\theta_j^* - \theta_{GT,j}\|_F^2$, constraining the 24 joint rotation matrices via the Frobenius norm.

The balancing weights are configured as $\lambda_\theta = 1.0$ and $\lambda_{vertex} = \lambda_J = \lambda_{J_{SMPL}} = 5.0$.

Key Experimental Results¶

Main Results¶

The model is quantitatively evaluated on three public benchmark datasets: LiDARHuman26M, SLOPER4D, and Human-M3. Evaluation metrics include MPJPE (mm), PA-MPJPE (mm), PVE (mm), and PCK thresholds PCK-30 and PCK-50 (%).

Dataset	Method	MPJPE (mm)↓	PA-MPJPE (mm)↓	PVE (mm)↓	PCK-30 (%)↑	PCK-50 (%)↑
LiDARHuman26M	LiDARCap† (CVPR'22)	93.74	76.96	120.14	82.08	92.27
	LiDARCap (Sequential)	79.31	66.72	101.64	86.00	95.00
	VoteHMR (ACM MM'21)	133.48	108.74	164.72	70.52	84.89
	LiDAR-HMR (TMM'25)	76.07	67.14	101.73	86.12	94.75
	SynHMR (Ours)	74.25	65.04	96.22	87.26	95.39
SLOPER4D	VoteHMR (ACM MM'21)	54.02	41.51	63.51	93.90	98.15
	LiDARCap (Sequential)	101.89	78.93	122.35	78.15	89.77
	LiDAR-HMR (TMM'25)	47.78	36.30	49.75	94.90	98.35
	SynHMR (Ours)	34.09	26.87	41.32	97.03	99.02
Human-M3	LiDARCap† (CVPR'22)	101.28	67.64	115.32	78.37	89.58
	LiDAR-HMR (TMM'25)	76.35	56.53	86.92	87.63	94.78
	SynHMR (Ours)	56.76	43.03	70.70	93.10	97.15

Ablation Study¶

Ablations on LiDARHuman26M isolate the contributions of key components, vertex sampling counts $N_v$, and decoder layers $L$.

Config	Joint Queries	Mesh Queries	GFPE	NAL	MPJPE (mm)↓	PVE (mm)↓	Description
Baseline (LiDARCap† equiv.)	-	-	-	-	93.74	120.14	PointNet++ with decoupled MLP & GCN
Joint Queries Only	✓	-	-	-	79.84	102.49	Transformer refines joints without surface mesh
Synergistic Joint-Mesh	✓	✓	-	-	75.89	98.33	Bidirectional co-refinement between joint & vertex
Joint + GFPE	✓	-	✓	-	78.09	100.78	High-frequency encoding on joint queries
Joint + NAL	✓	-	-	✓	78.30	100.92	Noise-augmented training on joints only
Synergistic + GFPE	✓	✓	✓	-	75.62	98.12	Synergistic queries with GFPE
Synergistic + NAL	✓	✓	-	✓	75.09	96.97	Synergistic queries with noise injection
SynHMR Full Model	✓	✓	✓	✓	74.25	96.22	Complete architecture, 19.49 mm MPJPE gain

Sampled Vertices $N_v$	MPJPE (mm)↓	PVE (mm)↓	Observation
32	75.21	97.38	Sparse surface sampling limits geometric anchoring
64	74.92	97.07	Progressive gain as coverage expands
128	74.75	96.72	Steady improvement
256 (Default)	74.25	96.22	Optimal trade-off between precision and memory
512	74.21	96.00	Marginal improvement with higher computation cost

Efficiency Comparison	Latency (ms)↓	GFLOPs↓	Params (M)↓	MPJPE (mm)↓	PVE (mm)↓
VoteHMR (ACM MM'21)	18.88	4.35	1.78	133.48	164.72
LiDAR-HMR (TMM'25)	55.28	13.94	46.40	76.07	101.73
LiDARCap (CVPR'22)	8.62	9.18	34.93	79.31	101.64
SynHMR (Ours)	7.56	3.87	14.80	74.25	96.22

Key Findings¶

Bidirectional synergy drives primary performance gains: Transitioning from decoupled joint queries to synergistic joint-mesh queries cuts MPJPE from 79.84 mm to 75.89 mm and PVE from 102.49 mm to 98.33 mm. This demonstrates that incorporating explicit surface queries provides critical geometric bounding constraints that prevent limb joints from drifting in sparse point clouds.
Noise injection acts as an effective regularizer: Introducing NAL during training yields a substantial drop in MPJPE (from 75.89 mm to 75.09 mm in synergistic configurations) without incurring any inference overhead, confirming that simulated joint noise trains the model to actively rely on surface observations for self-correction.
Superior real-time efficiency: SynHMR executes inference in only 7.56 ms with 3.87 GFLOPs and 14.80 M parameters. Compared to the previous leading single-frame method LiDAR-HMR (55.28 ms, 13.94 GFLOPs), SynHMR is over 7.3× faster, comfortably enabling real-time deployment at >100 FPS on automotive sensing platforms.

Highlights & Insights¶

Symmetric joint-mesh co-refinement: The paper departs from the conventional rigid hierarchy of "estimate pose first, fit mesh second", establishing a symmetric bidirectional query architecture where local surface geometry and global kinematic structure mutually reinforce each other.
Noise-Augmented Learning (NAL) for zero-overhead robustness: Injecting controlled Gaussian noise during training acts as a denoising proxy that enforces bidirectional dependencies between joints and vertices, delivering significant accuracy gains while keeping inference strictly unchanged.
Ultra-lightweight architecture with SOTA precision: By operating over sparse query sets ($N_v=256$, 24 joints) and leveraging PointNet++ with a Spatial GCN, the framework achieves top-tier reconstruction metrics while requiring less than 4 GFLOPs.

Limitations & Future Work¶

Lack of temporal modeling: As a single-frame pipeline, the method does not exploit temporal motion priors across consecutive LiDAR frames, which can lead to minor frame-to-frame jitter during rapid sports activities.
Sensitivity at extreme ranges (>40 m): In ultra-long-range scenarios where an entire human body may consist of fewer than 20 points, the coarse initial MLP predictions may fail, reducing the corrective effectiveness of subsequent Transformer layers.
Future directions: Integrating lightweight temporal filters or recurrent state-space models (e.g., Mamba) to leverage sequence continuity while preserving low latency.

vs LiDAR-HMR (TMM'25): LiDAR-HMR employs a coarse-to-fine progressive framework but still preserves a unidirectional pose-to-mesh dependency, suffering from a heavy computational latency of 55.28 ms; SynHMR uses symmetric query interactions to achieve a 13.69 mm MPJPE reduction on SLOPER4D while running 7.3× faster.
vs LiDARCap (CVPR'22): LiDARCap relies on temporal sequence features coupled with inverse kinematics (IK) and optimization-based SMPL fitting; SynHMR's pure feed-forward single-frame network decisively outperforms LiDARCap's multi-frame version (74.25 mm vs 79.31 mm on LiDARHuman26M) while eliminating iterative fitting overhead.
vs VoteHMR (ACM MM'21): VoteHMR uses point-level voting to aggregate joint features, which suffers severe divergence under sparse outdoor LiDAR point clouds (MPJPE > 130 mm); SynHMR anchors vertices via structured FPS sampling and an anatomical GCN, maintaining biomechanical fidelity even with sparse points.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Conceptually breaks the decoupled pose-mesh pipeline via a symmetric bidirectional query Transformer and noise-augmented learning.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across three public benchmarks (LiDARHuman26M, SLOPER4D, Human-M3) with detailed ablations on queries, sampling density, and runtime efficiency.
Writing Quality: ⭐⭐⭐⭐⭐ Clear problem statement, rigorous mathematical formulation, and well-structured qualitative and quantitative analyses.
Value: ⭐⭐⭐⭐⭐ Outstanding balance of reconstruction accuracy and runtime efficiency (7.56 ms latency, 3.87 GFLOPs), holding strong practical relevance for autonomous driving and mobile robotics.