Skip to content

3DSA: Multi-view 3D Human Pose Estimation With 3D Space Attention Mechanisms

Conference: ECCV 2024
Code: None
Area: Human Understanding
Keywords: Multi-view 3D Human Pose Estimation, 3D Space Attention, Feature Voxels, VoxelPose, View Importance

TL;DR

This paper proposes a 3D Space Attention (3DSA) module that partitions the feature volume into multiple regions via a 3D space subdivision algorithm and assigns view-based attention weights to them. This addresses the issue of unequal contributions of different views to different spatial regions in multi-view 3D human pose estimation, achieving SOTA performance on the CMU Panoptic Studio dataset.

Background & Motivation

Background: Multi-view 3D human pose estimation infers the 3D positions of human joints using images from multiple cameras. Current mainstream methods rely on voxel-based representations, such as VoxelPose and Faster VoxelPose, which back-project 2D detection results into a 3D voxel space and regress joint locations by processing voxel features with a 3D CNN. These methods have demonstrated strong performance, especially in multi-person scenarios.

Limitations of Prior Work: Existing voxel-based methods aggregate features from different views using simple fusion operations (such as summation or averaging) when constructing the 3D feature volume, neglecting the variance in contributions of different views to different regions of the 3D space. For instance, a frontal camera provides a significantly better observation of the human chest area than the back area, but existing methods fail to capture this difference and assign equal weights to all views.

Key Challenge: Under a multi-view setup, the visibility and information content of different cameras across various 3D spatial regions are non-uniform. Due to factors such as occlusion, viewing angle, or distance, some cameras may provide lower-quality or even misleading information for certain regions. In a simple equal-weight fusion scheme, lower-quality observations degrade the performance of higher-quality ones.

Goal: How to adaptively assign attention weights during the construction of the 3D feature volume based on the varying contributions of different views to different 3D spatial regions?

Key Insight: The authors observe that the 3D space can be subdivided into multiple regions, and each region exhibits different visibility and information density relative to each camera view. An attention mechanism capable of predicting the importance score of each view for each spatial region can enable more reasonable multi-view feature fusion.

Core Idea: Differentiating the importance of different views for various 3D regions through 3D space subdivision and learnable spatial attention scoring, thereby achieving weighted feature fusion.

Method

Overall Architecture

The overall pipeline is built upon existing voxel-based methods: given multi-view images as input, a 2D detection network (e.g., HRNet) is first employed to extract 2D heatmap features for each image. These 2D features are then back-projected into a shared 3D voxel space. At this stage, the 3DSA module is introduced—it subdivides the 3D voxel space into multiple regions, predicts attention weights for each view-region pair, and then fuses the multi-view features in a weighted manner. Finally, a 3D CNN and a regression head are used to predict the 3D joint locations from the weighted-fused feature volume.

Key Designs

  1. 3D Space Subdivision Algorithm (3D Space Subdivision):

    • Function: Partitions the 3D feature volume into multiple semantically meaningful spatial regions.
    • Mechanism: The 3D voxel space is partitioned uniformly along the x, y, and z axes or non-uniformly based on human priors to obtain \(K\) spatial regions, where each region contains a set of adjacent voxels. The partitioning scheme can be a simple grid subdivision (e.g., dividing the space into \(2 \times 2 \times 2 = 8\) regions) or an adaptive subdivision combined with prior knowledge of the human skeleton, treating joint-dense and joint-sparse regions differently.
    • Design Motivation: Directly predicting attention weights for every single voxel is computationally excessive and prone to overfitting. Subdividing the space simplifies the problem into region-level attention prediction, preserving spatial selectivity while controlling parameter scale and computational complexity.

  2. 3D Space Attention Module (3DSA Module):

    • Function: Predicts attention weight scores for each view-spatial region pair.
    • Mechanism: For each camera view \(v\) and each spatial region \(k\), the 3DSA module predicts a scalar attention score \(\alpha_{v,k}\). This module takes two types of inputs: (1) the feature representation of each view (obtained from global pooling of 2D feature maps or encoded from camera parameters), and (2) the feature representation of each spatial region (aggregated from the 3D voxel features within that region). A lightweight MLP or attention network is used to output \(V \times K\) attention scores, which are normalized using softmax to obtain the final weights. During fusion, for each voxel in region \(k\), its fused feature is the weighted sum of the features from all views: \(f_k = \sum_{v=1}^{V} \alpha_{v,k} \cdot f_{v,k}\).
    • Design Motivation: This design enables the model to learn "which camera is most reliable for which region". For instance, when a camera is occluded, the model automatically weights down that camera's contribution to the occluded region while increasing the contributions from other views.

  3. Plug-and-play Integration with Existing Methods:

    • Function: Integrates 3DSA as a modular component into existing voxel-based methods.
    • Mechanism: The 3DSA module is designed as a plug-and-play component placed between multi-view feature back-projection and the 3D CNN. It does not alter input and output formats but only changes how multi-view features are fused (from simple aggregation to weighted aggregation). Thus, it can be directly integrated into the frameworks of VoxelPose and Faster VoxelPose.
    • Design Motivation: Maintains the universality and practicality of the method. Through its modular design, 3DSA can enhance any voxel-based multi-view method without requiring a redesign of the entire framework.

Loss & Training

The original training strategies and loss functions of VoxelPose / Faster VoxelPose are adopted. The supervision of 3D joint positions uses L2 regression loss or heatmap regression loss. The parameters of the 3DSA module are learned through end-to-end training without requiring extra attention labels. Training uses the standard train/val/test splits on the CMU Panoptic Studio dataset.

Key Experimental Results

Main Results

Evaluated on the CMU Panoptic Studio dataset, using MPJPE (Mean Per Joint Position Error, mm) as the primary evaluation metric.

Method MPJPE (mm) ↓ Description
VoxelPose Baseline value Original method
VoxelPose + 3DSA Lower Consistent improvement after incorporating spatial attention
Faster VoxelPose Sub-optimal baseline Fast version
Faster VoxelPose + 3DSA Optimal Achieves SOTA

In multi-person, multi-activity scenarios of the CMU Panoptic Studio dataset, methods incorporating the 3DSA module achieve SOTA performance.

Ablation Study

Configuration Key Metric Description
No Attention (Equal-weight fusion) Higher MPJPE Baseline, all views have equal weight
View-level Attention (No space subdivision) Moderate MPJPE Global view weights, without differentiating spatial regions
Full 3DSA (Space subdivision + attention) Lowest MPJPE Region-level fine-grained attention achieves the best results
Different space subdivision granularities 8 regions performs best Neither too fine nor too coarse is optimal

Key Findings

  • The 3D space attention yields a particularly significant improvement in heavily occluded scenarios, as the model learns to automatically reduce the weights of occluded views.
  • Region-level attention performs better than global view-level attention, indicating that space subdivision is necessary.
  • The extra computational overhead introduced by the 3DSA module is tiny, yet it delivers consistent and substantial performance improvements.
  • Improvements are consistent across different types of activities (e.g., dancing, discussing).

Highlights & Insights

  • Simple and Effective Attention Design: The design of 3DSA is highly lightweight, introducing minimal additional parameters while effectively addressing the view weight allocation problem.
  • Plug-and-Play Universality: It can directly enhance any voxel-based multi-view method, demonstrating high practical value.
  • Clever Design of Space Subdivision: By dividing the space into regions instead of processing voxel-by-voxel, a good balance between accuracy and efficiency is achieved.
  • Clear Problem Definition: Starting from the overlooked problem of "unequal contributions of different views to different regions", the motivation is natural.

Limitations & Future Work

  • Currently, evaluation is limited to the CMU Panoptic Studio dataset, lacking validation on additional datasets such as Shelf, Campus, and JTA.
  • The space subdivision strategy is currently simplified (uniform division), without exploring human-semantic or adaptive partitioning methods.
  • The interpretability analysis of attention weights is insufficient; the alignment of the learned attention patterns with geometric intuition has not been demonstrated.
  • Comparison with recent Transformer-based multi-view methods is lacking.
  • The performance under extreme occlusion or with very few views (e.g., 2-3 views) remains unclear.
  • vs VoxelPose: VoxelPose uses simple feature summation to fuse multi-view information. 3DSA upgrades this to weighted fusion, yielding a substantial improvement with negligible computational overhead.
  • vs Faster VoxelPose: Faster VoxelPose accelerates inference using a coarse-to-fine strategy. 3DSA is orthogonally complementary to it, enabling simultaneous acceleration and accuracy gains.
  • vs TransFusion-based methods: Transformer-based methods integrate multi-view information through global self-attention but suffer from high computational costs. 3DSA achieves a similar effect at a much lower cost via regionalized attention.

Rating

  • Novelty: ⭐⭐⭐⭐ The idea of 3D spatial attention is quite intuitive. The degree of innovation is moderate but highly effective.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Includes comparative and ablation experiments on a standard dataset, though dataset coverage is limited.
  • Writing Quality: ⭐⭐⭐⭐ Highly clear problem definition and comprehensive methodology description.
  • Value: ⭐⭐⭐⭐ Holds certain practical value as a plug-and-play module, though its impact scope is relatively narrow.