Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Video

Conference: ECCV 2026
arXiv: 2512.12165
Code: None
Area: 3D Vision
Keywords: passive scene sounds, camera pose estimation, spatial audio, multimodal fusion, direction of arrival (DOA)

TL;DR

This paper proposes the first method that leverages passive scene audio to assist vision-based relative camera pose estimation. By performing late fusion of a spatial audio encoder (SAE)—composed of a direction-of-arrival (DOA) spectrum and a binauralized embedding—with the SOTA visual pose estimation model Reloc3r, this method achieves significant improvements in both real-world scenes and simulated datasets.

Background & Motivation

Relative camera pose estimation is a core problem in embodied perception and 3D scene understanding. Vision-based methods (such as SuperGlue, LoFTR, DUSt3R, and Reloc3r) have made significant progress over the past few years, but these methods still degrade severely in visually degraded scenarios, such as motion blur, low lighting, occlusion, and lack of texture. In the real world, videos recorded by wearable cameras (e.g., dim concert venues, fast-moving ball games) frequently exhibit such degradations, making it difficult to stably recover camera motion solely relying on visual signals. Conversely, prior works attempting audio-assisted spatial localization either rely on active sonar (such as biomimetic echolocation in BatVision) or operate entirely in simulation for sound source localization and navigation; none has leveraged "sounds inherently present in the scene" in real-world, in-the-wild videos to help recover the 6-DoF relative camera pose.

Sound naturally possesses geometric properties that are unaffected by lighting and occlusion. As the camera moves through a scene, multichannel microphone-captured audio signals undergo predictable changes in spatial attributes—the directional distribution of sound sources shifts with camera rotation, left-right channel differences vary with camera translation, and even reverberation patterns change with position. These spatial audio cues are "free": consumer devices (GoPro, Insta360 GO, Ray-Ban Meta glasses, Project Aria glasses) are already widely equipped with dual-channel or even seven-channel microphones, naturally attaching multichannel audio streams to every video. The challenge lies in how to extract learnable spatial features related to camera motion from raw audio and fuse them effectively with visual features—preventing uninformative audio from degrading strong visual performance while ensuring audio can compensate when vision fails.

The key insight of this paper is to build a dual-pathway spatial audio encoder: one pathway utilizes the analytical MUSIC++ algorithm to directly calculate the sound direction of arrival (DOA) spectrum, explicitly capturing the azimuthal distribution of dominant sound sources; the other pathway obtains binauralized embeddings through a novel view acoustic synthesis (NVAS) pretraining task, implicitly encoding the spatial structure of the entire sound field. The features from both pathways are concatenated and late-fused with the visual features of Reloc3r. Core Idea: To utilize a spatial audio encoder with complementary analytical DOA spectra and learned binauralized embeddings, integrated into a SOTA visual pose estimation backbone via late fusion, providing robust geometric cues from passive scene sounds for camera pose estimation without interfering with strong visual signals, especially compensating for performance loss during visual degradation.

Method

Overall Architecture

The proposed method consists of two core modules: a Spatial Audio Encoder (SAE) and an audio-visual relative pose predictor. Given a pair of source and target RGB images \((I_s,I_t)\) and their synchronized multichannel audio clips \((A_s,A_t)\), the SAE extracts spatial features from the audio, which are then concatenated with the visual features from the Reloc3r decoder at the feature level and fed into a pose regression head to predict the bidirectional relative 6-DoF poses \(P_{s\to t}\) and \(P_{t\to s}\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 440}}}%%
flowchart TD
    S["Source Image $I_s$"] --> V["Visual Encoder<br/>Reloc3r E-D"]
    T["Target Image $I_t$"] --> V
    V --> FV["Visual Feature $f^V$"]

    SA["Source Audio $A_s$"] --> SAE["Spatial Audio Encoder SAE"]
    TA["Target Audio $A_t$"] --> SAE

    SAE --> |"Concatenation & Linear Projection"| FA["Spatial Audio Feature $f^A$<br/>(768-dim)"]

    FV --> FUSE["Concatenation<br/>$f^{AV}{=}[f^A||f^V]$"]
    FA --> FUSE
    FUSE --> HEAD["Pose Regression Head<br/>(4 linear layers)"]
    HEAD --> P1["$P_{s\\to t}$"]
    HEAD --> P2["$P_{t\\to s}$"]

    subgraph SAE_INT["Inside SAE"]
        direction TB
        IN["Multichannel STFT"] --> D1["MUSIC++ DOA<br/>$f^D \\in [0,1]^{360}$"]
        IN --> D2["Binaural Feature Extractor<br/>$f^B \\in \\mathbb{R}^{1024}$"]
        D1 --> C["Concatenation $f^A{=}[f^D||f^B]$"]
        D2 --> C
    end

Key Designs

1. Dual-Pathway Spatial Audio Encoding: Analytical DOA spectra explicitly encode sound source azimuth, while binauralized embeddings implicitly model the full sound field

The audio signal itself does not directly dictate which direction the camera is moving, making the feature extraction approach critical. Starting with the multichannel STFT spectrogram \(S_j \in \mathbb{R}^{C\times F\times W}\) of the multichannel audio, the SAE computes two complementary representations in parallel.

The first pathway is the direction of arrival (DOA) spectrum. An analytical 360-dimensional vector \(f^D \in [0,1]^{360}\) is calculated from the multichannel spectrum using the MUSIC++ algorithm (MUSIC with frequency normalization), where each element corresponds to the average audio energy within a \(1^\circ\) azimuthal sector. The DOA spectrum intuitively demonstrates the distribution of main sound sources relative to the microphone—when the camera rotates, the peak locations shift accordingly, providing a direct geometric constraint for rotation estimation. This pathway is completely analytical, requires no learnable parameters, and remains effective in multi-source scenes. The second pathway is a learned binauralized embedding \(f^B \in \mathbb{R}^{1024}\), extracted from a pre-trained NVAS model encoder trained on the large-scale Ego-Exo4D dataset. Unlike the DOA spectrum which only captures the direction of dominant sound sources, the binauralized embedding implicitly encodes the spatial structure of the entire sound field—reverberation patterns, the spatial arrangement of multiple overlapping sound sources, and the distribution of ambient noise—which are particularly valuable in uniform environments or with multiple overlapping sources. The intuition behind the complementarity of the two pathways is that the DOA spectrum is proficient in localizing discrete dominant sound sources but becomes flat and uninformative in uniformly diffused noise or heavy sound overlaps; conversely, the binauralized embedding senses fine-grained textures of the overall sound field but lacks precision for targeting individual sources. Concatenating both into \(f^A = [f^D || f^B]\) and projecting via a linear layer to 768 dimensions yields a spatial audio representation capable of both "accurate localization" and "global modeling."

2. Novel View Acoustic Synthesis (NVAS) Pretraining: Forcing the encoder to learn spatial geometry via a "sound viewpoint transformation" task

The quality of the binauralized embedding depends on the design of the pretraining task. This paper proposes the Novel View Acoustic Synthesis (NVAS) task: given the binaural audio of the source view and the monaural audio of the target view, the model is required to synthesize the binaural audio of the target view. The key to this task is that, to accurately predict how the sound would hear when shifting from the source microphone position to the target microphone position, the encoder must extract latent geometric cues closely related to the spatial positions of the microphones from the input audio.

During training, the model predicts the difference between the left and right channel spectrograms of the target audio instead of the full left and right channel spectrograms:

\[ \mathcal{L}^{B} = \|\tilde{S}^{\Delta}_{\mathcal{T}} - S^{\Delta}_{\mathcal{T}}\|_1, \quad S^{\Delta}_{\mathcal{T}} = \text{STFT}(A^{l}_{\mathcal{T}} - A^{r}_{\mathcal{T}}) \]

The strategy of predicting only the difference drastically reduces the output space dimensionality and accelerates training, while forcing the model to focus on encoding spatial information conveyed by the left-right channel differences. Once pretraining is complete, the encoder weights are frozen and used as a feature extractor, bypassing joint training for pose estimation. This two-stage strategy of "first learning spatial audio representations, then learning pose regression" ensures the generality of the embedding while lowering end-to-end training computation overhead.

3. Late Fusion: Protecting Vision, letting audio function only when needed

The key tension in multimodal fusion lies in the fact that audio is sometimes highly valuable (e.g., during visual degradation) and sometimes completely useless (e.g., in silent scenes). This paper adopts a minimalist late fusion strategy: directly concatenating the spatial audio features of the SAE with the visual features output by the Reloc3r decoder at the feature level before feeding them to the pose regression head. Specifically, the visual features and audio features are projected to 768 dimensions through independent linear layers, then concatenated as \(f^{AV} = [f^A || f^V]\).

The core motivation behind this design is to "protect vision": prior to fusion, visual feature extraction is completed independently, unaffected by the audio signals. When the audio signal is uninformative (such as in complete silence) or unrelated to camera movement, the regression head can adaptively scale down the weight of the audio dimensionality without interfering with the already strong visual features. Experiments validate the effectiveness of this design—under good visual conditions, incorporating audio does not degrade performance, and indeed still offers a slight improvement (outperforming the vision-only baseline in 95% of cases); under visual degradation, audio geometric cues effectively back up the system, causing the relative gain of total pose AUC@5 to jump from 42% under normal conditions to 67% under degraded conditions. Compared to feature-level interaction fusion schemes like cross-attention, late fusion is simpler, more stable during training, and more robust to modal absence (e.g., videos without audio) in practical deployment.

Loss & Training

The pose prediction loss adopts the design of Reloc3r, containing both rotation and translation components, averaged across the source-to-target and target-to-source directions:

\[ \mathcal{L}^{P} = L^{P}_{s \to t} + L^{P}_{t \to s} \]

Training hyperparameters: 100 epochs (including 5 epochs of linear warmup), initial learning rate of \(10^{-5}\) cosine-annealed to \(10^{-7}\), batch size of 64. The visual branch uses the Reloc3r-512 version: input images of size 256×256, 24-layer encoder + 12-layer decoder. Audio parameters: sampling rate 48kHz, clip length 1000ms, STFT FFT bin size 1024, spectrogram size 512×96. Training utilizes 8 NVIDIA A40 GPUs (48GB VRAM).

Key Experimental Results

Main Results (Ego-Exo4D Validation Set)

Metric	Method	AUC@5	AUC@10	AUC@20
Rotation	Vision Only (Reloc3r)	37.35	53.62	68.98
Rotation	Ours	38.54	54.49	69.61
Translation	Vision Only (Reloc3r)	0.73	2.63	7.77
Translation	Ours	1.03	3.36	9.34
Total Pose	Vision Only (Reloc3r)	0.57	2.33	7.33
Total Pose	Reloc3r-AV Cross-Att	0.76	2.83	8.34
Total Pose	Ours	0.81	2.99	8.82
Total Pose	Relative gain vs Vision Only	+42.1%	+28.3%	+20.3%

Robustness under Visual Degradation (Ego-Exo4D)

Method	AUC@5	AUC@10	AUC@20
Vision Only (Degraded)	0.21	1.00	3.66
Reloc3r-AV (Degraded)	0.29	1.27	4.36
Ours (Degraded)	0.35	1.44	4.80
Relative gain vs Vision Only	+66.7%	+44.0%	+31.2%

HM3D-SS Simulator Dataset

Method	Rotation MAE (°) Original	Rotation MAE (°) Degraded
SLfM [slfm]	0.77	—
Vision Only (Reloc3r)	0.09	3.83
Ours	0.06	2.46
Relative gain vs Vision Only	+33.3%	+35.8%

Ablation Study (Ego-Exo4D Total Pose AUC)

Config	AUC@5	AUC@10	AUC@20	Note
Ours (Full)	0.81	2.99	8.82	full model
w/o DOA	0.75	2.86	8.41	w/o DOA spectrum
w/o Binaural	0.74	2.80	8.34	w/o Binaural embedding
Monaural substitution	0.74	2.82	8.37	monaural replacing binaural
Vision Only	0.57	2.33	7.33	baseline

Key Findings

• The gain in translation estimation is significantly larger than in rotation: The relative gain in total pose AUC@5 reaches +42.1% (translation +41.1% vs. rotation +3.2%), indicating that spatial audio cues provide a much more direct geometric constraint for 'how much the camera displaced' compared to rotation—the translation sensitivity of the DOA spectrum and the acoustic distance cues of the binauralized embedding are both positively correlated with displacement magnitude.
• Dual-pathway features are complementary rather than redundant: Separately removing the DOA or binauralized embedding both cause a decline in AUC@5 by 0.06-0.07, demonstrating that both contribute irreplaceable information. DOA is stronger for rotation estimation (a prominent drop in rotation AUC is observed after removing DOA), while binauralization is more useful for translation and in complex sound fields.
• The audio compensation effect scales up during visual degradation: The relative improvement of AUC@5 jumps from 42% to 67%, validating the late-fusion design goal of 'audio actively compensating when vision fails'. Even under the most severe degradation conditions (Gaussian blur \(\sigma=8\)), the total pose AUC@20 of the proposed method (3.47) still exceeds that of the vision-only baseline (2.63).
• Geometric signaling rather than semantic correlation of audio: In negative control experiments (synchronous audio -> 1-5s shift -> in-scene shuffle -> cross-scene substitution step-by-step collapse), performance progressively degrades to lower than the visual baseline (AUC@5 drops from 0.81 to 0.50, which is < 0.57 of Vision Only). This indicates that the model relies on the geometric consistency of the audio with the motion rather than dataset-specific semantic patterns.
• Audio characteristics analysis: Audio provides the greatest help in scenarios containing multiple sources or mixed far-field and near-field situations (Soccer 84.6%, Music 81.4%); its contribution in single dominant source or silent environments is limited but does not drag down performance.

Highlights & Insights

• 'Passive audio' pioneers a new paradigm for pose estimation: Prior related works either used active sonar or performed source localization in simulation—this work is the first to prove that passive scene sounds can help recover 6-DoF camera poses in real-world, in-the-wild videos. This approach of 'leveraging accompanying signals to assist spatial perception' can be generalized to other multimodal perception tasks.
• The analytical + learned dual-pathway design is an extremely efficient and practical solution: The DOA pathway uses the analytical MUSIC++ algorithm, requiring almost no parameters; the binauralized embedding can be frozen and reused after a single pretraining phase. The output dimensionality of both pathways is only 1384 (360+1024), which is concatenated with visual features after a single linear projection, keeping parameter counts very small. Compared to end-to-end multimodal Transformers, this hybrid architecture holds advantages in both computational and data efficiency.
• The 'protecting vision' philosophy of late fusion is worth popularizing: In multimodal systems, the greatest risk is the 'strong modality being skewed by the weaker modality.' The proposed strategy allows each modality to complete extraction independently before concatenation, enabling the regression head to naturally learn the contribution weights of each modality. Simple, stable to train, and robust to modal absence, it represents a more pragmatic choice than complex interaction schemes like cross-attention for real-world deployment.
• Negative control experiments are the gold standard for validating geometric signs: Mapping out a smooth performance decline curve across four degradation levels (synchronization / temporal shift / in-scene shuffle / cross-scene substitution) powerfully rebuts the potential criticism that 'the model is merely learning scene-semantic associations'—this demonstration strategy is highly recommended for other multimodal works.

Limitations & Future Work

• The absolute value of translation estimation remains low: AUC@5 is 0.81, indicating that in most test samples, the predicted pose error exceeds the 5-degree threshold. The authors point out that translation recovery remains a bottleneck during long-range displacement or complex acoustic scenes.
• Sensitivity to sudden dominant sound sources: When a loud sound source suddenly appears or disappears between two frames (e.g., a musician only starting to play in the second frame), the model may be misled by the audio signal. This is an inherent vulnerability of current passive audio schemes where the audio 'lies' and the model cannot easily verify it.
• Dependence on multichannel microphone hardware support: The method requires at least dual-channel audio to extract spatial features. Monaural ablation experiments show performance degrading close to the vision-only baseline. However, the development of real-time binaural upmixing (mono-to-binaural) technologies might alleviate this constraint.
• Unverified cross-device/cross-dataset generalization: Training and testing were both performed on the Project Aria glasses (7-channel mic) config, and the model's generalizability to other microphone arrays (such as GoPro 2-channel mic) has not been experimentally validated.
• Future directions: Introducing an audio quality estimation gating module to dynamically adjust fusion weights based on the sharpness of the DOA spectrum or the confidence of the binauralized embedding; combining IMU signals (supplementary experiments have shown that audio and IMU are complementary); considering longer audio context windows to capture continuous acoustic field evolution.

Related Work & Insights

• vs SLfM [slfm]: SLfM jointly learns sound source localization and camera motion but is trained and tested entirely in simulation, requiring ground truth DOA supervision. This work validates on real-world video (Ego-Exo4D) without DOA annotations, employing NVAS pretraining to replace the SLfM joint training paradigm.
• vs Reloc3r [reloc3r]: Reloc3r is a pure-vision SOTA pose estimation model on which this work builds an audio branch. It performs slightly better under normal vision and leads significantly under degraded conditions (AUC@5 rises from 0.21 to 0.35, +67%).
• vs DUSt3R / MonST3R: These end-to-end Transformer dense correspondence methods do not involve the audio modality and are orthogonal to this work. Interestingly, the late fusion design of this paper is fully compatible with replacing the visual backbone—swapping Reloc3r for DUSt3R for visual feature extraction should be equally effective.
• vs Audio-Visual Self-Supervised Learning (L3-Net, AV-MAE, etc.): Previous self-supervised audio-visual methods focus on semantic-level cross-modal alignment or event classification, whereas this work applies audio to geometric spatial reasoning, differing completely in task nature and feature requirements.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First to introduce passive scene sounds to relative camera pose estimation in real-world videos, establishing a new problem definition and solution of 'assisting spatial perception with cheap accompanying signals'.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across both real-world and simulated datasets (including visually degraded versions), featuring comprehensive ablation designs (DOA / Binaural / Monaural / Cross-Attention comparisons), highly convincing negative control experiments, and in-depth acoustic characteristics analysis (qualitative + quantitative across 8 scene categories).
• Writing Quality: ⭐⭐⭐⭐⭐ The motivation starts infectiously from a specific scene (a dimly lit concert), and the methodology is presented clearly with a distinct hierarchy. Extensive analytical experiments (Tab.4 performance decomposition under various acoustic conditions) provide readers with a deep understanding of 'when and why audio is helpful'.
• Value: ⭐⭐⭐⭐⭐ Transitions the practical direction of 'cheap accompanying signals -> spatial geometric reasoning' into real-world scenarios, offering direct implementation insights for AR/VR glasses, embodied AI hardware, and other devices naturally equipped with multichannel microphones. Although the absolute translation accuracy does not yet reach the deployment threshold, the problem definition and solution design are highly inspiring.

Related Papers

• [ECCV 2026] Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation
• [CVPR 2026] SonoWorld: From One Image to a 3D Audio-Visual Scene
• [CVPR 2026] WildPose: A Unified Framework for Robust Pose Estimation in the Wild
• [CVPR 2026] ORBIT: Benchmarking SfM in the Wild with 360° Video
• [CVPR 2025] Extreme Rotation Estimation in the Wild