Sound-based Multi-Person 3D Pose Estimation

Conference: ECCV 2026
Paper: ECCV Official
Project: https://oumi03.github.io/sound-mhpe/
Area: Human Understanding
Keywords: active acoustic sensing, multi-person 3D pose estimation, spatio-temporal modeling, multi-scale STFT, DETR-based decoder

TL;DR

The paper presents the first attempt to estimate multi-person 3D poses solely from acoustic signals, introducing SoundMHPE which leverages an Acoustic Multi-scale Encoder for time-frequency disentanglement and a Temporal Pose Decoder with per-frame multi-queries to model intra-person dynamics and inter-person acoustic interactions.

Background & Motivation

Perceiving and tracking 3D human poses in complex indoor environments is a fundamental problem in computer vision, underpinning applications from healthcare monitoring to sports analysis and search-and-rescue operations. Conventional multi-person pose estimation relies heavily on RGB cameras or video feeds; however, optical approaches suffer severe degradation under low-light or pitch-black conditions, fail in the presence of visual line-of-sight occlusions, and raise persistent privacy concerns when deployed in private spaces such as bedrooms or bathrooms. While wireless radio-frequency technologies (e.g., WiFi CSI, mmWave, UWB radar) can penetrate certain walls and obstacles, RF signals are easily attenuated or blocked by conductive metals and water, and they cannot be legally deployed in restricted settings such as intensive care units or aircraft cabins. Active acoustic sensing—transmitting low-power structured acoustic signals and receiving their reflections via microphone arrays—operates unaffected by illumination, bypasses facial visual identification, and can diffract through narrow apertures where wireless frequencies struggle.

Despite these advantages, prior acoustic human pose estimation methods are strictly confined to single-person environments. Transitioning to multi-person scenarios introduces severe physical ambiguity into the received acoustic waveforms. When multiple subjects move concurrently in the same space, their motion-dependent acoustic perturbations superimpose linearly, preventing simple spatial or spectral separation of individual motion signatures. More critically, acoustic waves bounce between moving human bodies before reaching the receiver; these multi-body reflections introduce intricate, non-linear propagation delays that disrupt the direct temporal mapping between physical joint kinematics and received echo variations.

Existing single-person frameworks, which compress acoustic time-series into single static features, collapse under this multi-path superposition. Core idea: construct an encoder-decoder framework that explicitly decouples time-frequency resolutions on the encoding side via multi-scale STFT with orthogonal self-attention, while assigning frame-wise dedicated queries per subject on the decoding side to disentangle intra-person motion continuity from inter-person acoustic reflection dynamics.

Method

Overall Architecture

SoundMHPE employs an active acoustic sensing setup where a pair of loudspeakers emits periodic Time Stretched Pulse (TSP) signals and a four-channel Ambisonics microphone (\(W, X, Y, Z\)) records the reflected audio \(s\). Given an acoustic segment spanning \(N_{\text{in}} = N_{\text{out}} + N_{\text{prev}}\) frames of length \(L\), the network simultaneously predicts the 3D poses of \(M\) potential subjects across the subsequent \(N_{\text{out}}\) consecutive frames.

The pipeline comprises two core modules: the Acoustic Multi-scale Encoder (AME) and the Temporal Pose Decoder (TPD). First, AME extracts log-Mel spectrograms across multiple temporal window sizes to preserve both high-rate temporal transitions and fine frequency shifts, processing them with decoupled self-attention layers to produce an acoustic feature map. Next, TPD assigns \(N_{\text{out}} + 1\) learnable queries to each subject instance (covering \(N_{\text{out}}\) pose frames plus an instance-level confidence query). Through alternating intra-person motion attention and inter-person interaction attention, followed by cross-attention over the acoustic feature map, TPD isolates individual kinematic trajectories and outputs 3D joint coordinates alongside existence confidence scores.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["4-channel Ambisonics Signals<br/>W, X, Y, Z channels"] --> B["Multi-Scale STFT<br/>window sizes L, 2L, 4L"]
    B --> C["Acoustic Multi-scale Encoder<br/>AME Blocks (TSA + FSA)"]
    C --> D["Acoustic Feature Map"]
    E["Spatio-Temporal Queries<br/>M × (Nout + 1) learnable tokens"] --> F["Temporal Pose Decoder<br/>TPD Blocks (MSA + ISA)"]
    D --> G["Cross-Attention Conditioning"]
    F --> G
    G --> H["Bipartite Matching & Output<br/>Hungarian Loss & 3D Poses"]

Key Designs

1. Multi-scale STFT and time-frequency decoupled encoding: resolving temporal transience versus subtle Doppler shifts
A single fixed STFT window size imposes an unavoidable trade-off between time and frequency resolution in acoustic motion tracking. Short temporal windows precisely register rapid onset dynamics of limb movement but lack the frequency resolution needed to detect micro-Doppler signatures; long windows provide sharp frequency bins to isolate subtle body motions but smear temporal boundaries across multiple individuals. AME resolves this dilemma by executing multi-scale STFT with three distinct window durations: \(L\), \(2L\), and \(4L\), combined with \(B\), \(2B\), and \(4B\) Mel filter banks respectively. This produces three complementary representations: high-temporal-resolution \(a_{\text{high-time}} \in \mathbb{R}^{N_{\text{in}} \times C \times B}\), intermediate \(a_{\text{mid}} \in \mathbb{R}^{(N_{\text{in}}/2) \times C \times 2B}\), and high-frequency-resolution \(a_{\text{high-freq}} \in \mathbb{R}^{(N_{\text{in}}/4) \times C \times 4B}\). To prevent standard global attention from confusing multi-scale granularities, AME introduces two specialized self-attention steps: Temporal Self-Attention (TSA), which computes attention strictly within each individual resolution branch, and Frequency Self-Attention (FSA), which restricts attention across the 7 multi-scale tokens sharing the same \(4L\) physical time window (4 from \(a_{\text{high-time}}\), 2 from \(a_{\text{mid}}\), and 1 from \(a_{\text{high-freq}}\)), enabling local cross-spectral synergy without cross-temporal interference.

2. Per-frame multi-query pose decoding: mitigating temporal representation collapse
Standard DETR-like architectures dedicate a single query token to each potential object instance. When extended to sequential multi-frame estimation, compressing an entire multi-frame trajectory into a single embedding collapses temporal variance, depriving the cross-attention mechanism of the temporal granularity required to resolve time-delayed acoustic reflections. TPD overcomes this by introducing \(N_{\text{out}}\) separate pose queries per subject, augmented by 1 global instance query for confidence prediction, yielding \(M \times (N_{\text{out}} + 1)\) queries in total. Each pose query is explicitly conditioned on both a frame-specific temporal positional embedding and a subject-specific identity embedding. Consequently, earlier queries selectively attend to acoustic feature slices corresponding to earlier timestamps, while later queries focus on trailing echo responses. The instance query output passes through a linear layer to predict an existence probability \(\hat{c}_i\), allowing the model to suppress inactive subject slots during inference.

3. Decoupled motion and interaction attention: isolating individual biomechanics from multi-body acoustic crosstalk
With \(M \times (N_{\text{out}} + 1)\) queries active across the decoder, applying unconstrained global self-attention mixes physical joint continuity constraints with multi-person acoustic reflection artifacts. TPD decouples query self-attention into two orthogonal operators. Motion Self-Attention (MSA) confines attention computation to the \(N_{\text{out}} + 1\) queries belonging to the same individual subject, focusing purely on preserving temporal smoothness and biomechanically valid joint velocities. In contrast, Interaction Self-Attention (ISA) computes cross-subject attention between the \(i\)-th individual's queries and the \((M - 1) \times (N_{\text{out}} + 1)\) queries of all other subjects, explicitly guiding the network to capture mutual spatial displacement, acoustic occlusions, and inter-person sound scattering.

Loss & Training

During training, the Hungarian bipartite matching algorithm establishes optimal one-to-one pairings between the \(M\) predicted candidate instances and ground-truth human subjects. The total loss \(\mathcal{L}\) balances 3D geometric pose regression and instance classification:

\[\mathcal{L} = \mathcal{L}_{\text{pose}} + \lambda \mathcal{L}_c\]

where \(\mathcal{L}_{\text{pose}}\) is the mean squared error (MSE) between predicted and ground-truth 3D joint coordinates for matched instances, and \(\mathcal{L}_c\) is the binary cross-entropy (BCE) loss supervised by instance presence. The loss weight is set to \(\lambda = 0.5\). The model is optimized using AdamW with an initial learning rate of \(5 \times 10^{-5}\) and weight decay of \(1 \times 10^{-4}\) over 500 epochs.

Key Experimental Results

Main Results

The method was evaluated on the 6-hour Acoustic Multi-person Pose (AMP) dataset using a 3-fold cross-subject cross-validation protocol across 15 participants. Metrics include Mean Per Joint Position Error (MPJPE in mm), Procrustes-Aligned MPJPE (PA-MPJPE in mm), and Percentage of Correct Keypoints ([email protected]). Baselines include Adapted Shibata et al. (a single-person active acoustic pose estimator modified with a multi-person regression head) and Repurposed Yan et al. (the Person-in-WiFi 3D architecture adapted to receive acoustic spectrograms).

The main benchmark comparison from Table 2 of the paper:

Method	MPJPE [mm] (↓)	PA-MPJPE [mm] (↓)	[email protected] (↑)
Shibata et al. [24] (Adapted)	121.7	71.5	0.36
Yan et al. [30] (Repurposed)	119.9	69.7	0.36
SoundMHPE (Ours)	106.5	65.0	0.43

When broken down by subject count (Table 4 in the paper), SoundMHPE achieves an MPJPE of 95.0 mm in single-person settings (vs. 111.3 mm for Shibata et al.) and maintains 111.2 mm in challenging triple-person settings (vs. 124.5 mm for Shibata et al. and 122.4 mm for Yan et al.), demonstrating remarkable resistance to multi-person signal overlap.

Ablation Study

The paper validates the contribution of each architectural component through ablation experiments.

Component ablation results from Table 3 of the paper:

Config	MPJPE [mm] (↓)	PA-MPJPE [mm] (↓)	[email protected] (↑)	Note
Ours w/o AME	115.2	67.5	0.38	Single-scale STFT with standard self-attention
Ours w/o TPD	116.5	69.0	0.38	Single query per person predicting all frames
Ours (Full Model)	106.5	65.0	0.43	Full multi-scale encoder + per-frame decoder

Ablation on self-attention design (Table 5 in the paper) shows that replacing AME's TSA+FSA with standard self-attention degrades MPJPE to 111.7 mm, while replacing TPD's MSA+ISA with standard self-attention degrades MPJPE to 114.4 mm. Furthermore, STFT window evaluations (Table 6) confirm that the \((L, 2L, 4L)\) combination yields the lowest error (106.5 mm), whereas shifting to narrower windows \((L/4, L/2, L)\) increases error significantly to 118.5 mm.

Key Findings

• Removing TPD causes the largest performance drop (+10.0 mm MPJPE), proving that assigning independent queries to each time frame is critical for learning temporal acoustic alignments.
• Narrowing the STFT window to boost time resolution at the expense of frequency resolution degrades tracking accuracy, demonstrating that high spectral resolution is indispensable for isolating subtle motion signatures like arm raising and torso twisting.
• Applying SoundMHPE's decoupled attention and TPD architecture to WiFi CSI data on the Person-in-WiFi 3D benchmark (Table 7) reduces MPJPE from 127.4 mm to 122.6 mm and increases [email protected] from 0.13 to 0.31, proving the cross-modal versatility of the spatio-temporal decoupling framework.

Highlights & Insights

• Pioneers multi-person 3D human pose estimation solely via active acoustic sensing, expanding non-invasive human perception into dark, privacy-sensitive, and RF-restricted venues.
• Decouples multi-scale STFT processing into separate temporal (TSA) and frequency (FSA) attention mechanisms, circumventing the classical uncertainty barrier between transient timing and spectral resolution.
• Replaces standard single-token instance queries with frame-wise multi-queries in the DETR framework, achieving fine-grained temporal alignment between continuous physical kinematics and acoustic echo delays.

Limitations & Future Work

• The current evaluation was conducted primarily in a controlled indoor room with fixed acoustic boundaries; large changes in room geometry and wall acoustic absorption coefficients still pose generalization challenges.
• Transmitting periodic audible TSP chirp signals may cause acoustic distraction in practical settings; transitioning to inaudible ultrasound or embedding probe signals into background ambient music represents an important future direction.
• The experimental dataset spans up to three subjects; denser crowds and larger distances will require larger multi-channel microphone arrays to provide higher spatial angular resolution.

Related Work & Insights

• vs Shibata et al. [24] & Oumi et al. [18] (Single-Person Acoustic Pose): Prior works operate exclusively on single subjects and map whole spectrograms to single skeletons without addressing multi-person reflection delays; SoundMHPE reduces MPJPE by 13.3 mm in 3-person environments through multi-scale and temporal query decoupling.
• vs Yan et al. [30] (Person-in-WiFi 3D): WiFi methods use CSI for multi-person tracking but rely on single-frame queries that cannot model fine-grained multi-frame signal-pose correlations; when evaluated on WiFi CSI data, SoundMHPE's TPD temporal query structure improves PCKh from 0.13 to 0.31.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First framework demonstrating multi-person 3D pose estimation from acoustic echoes alone.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 6-hour synchronized dataset, thorough ablations, and cross-modal validation on WiFi data.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition linking physical acoustic properties to deep transformer design choices.
• Value: ⭐⭐⭐⭐☆ Opens a promising, privacy-preserving paradigm for unobtrusive multi-person tracking in complex environments.

Related Papers

• [ECCV 2026] CoDePose: Multi-View 3D Human Pose Estimation via Coupled 2D-3D Denoising Diffusion
• [ECCV 2026] DETRPose: Real-Time End-to-End Multi-Person Pose Estimation via Modified Transformer Decoder and Novel Denoising Keypoints
• [ECCV 2026] Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning
• [ECCV 2024] 3DSA: Multi-view 3D Human Pose Estimation With 3D Space Attention Mechanisms
• [ICLR 2026] From Sparse to Dense: Spatio-Temporal Fusion for Multi-View 3D Human Pose Estimation with DenseWarper