Whence the Voice? Self-supervised Dual-source Audio-Visual Localisation via Selective Convergence¶
Conference: ECCV 2026
Paper: ECCV 2026
Code: https://happy-new-bears.github.io/scav-project-page/
Area: Segmentation
Keywords: Audio-Visual Localisation, Dual-source, Selective Convergence, Self-supervised Learning, Pixel-level Benchmark
TL;DR¶
Discovering that contrastive audio-visual learning naturally exhibits selective convergence towards dominant sound sources in mixtures, this paper proposes SCAV, an annotation-free two-stage progressive decoupling framework, and introduces the large-scale pixel-level VGGSound-DuetMask benchmark.
Background & Motivation¶
In real-world environments, multiple sounding sources frequently sound concurrently. While humans easily navigate cocktail-party scenarios by selectively steering auditory attention to isolate individual sources, audio-visual machine perception confronts an inherent chicken-and-egg paradox: audio signals mix linearly through additive superposition in the acoustic domain, whereas visual scenes present sound-producing objects through disjoint spatial configurations. Decomposing the mixed audio requires spatial cues from the visual modality to determine source regions, yet identifying which visual regions produce sound conversely demands separated audio signals to establish clean cross-modal correspondences. This fundamental circular dependency renders joint multi-source localisation an ill-posed inverse problem where standard joint optimisation easily settles into trivial or collapsed representations without manual supervision.
Prior attempts to break this circular dependency have introduced complicated handcrafted heuristics or external supervision. Common strategies include building pseudo-class visual dictionaries via single-source unsupervised clustering, employing iterative curriculum learning with numerous hand-tuned rejection thresholds, directly incorporating ground-truth video category labels to supervise class tokens, or even relying on multimodal large language models (MLLMs) to produce rich descriptive prompts. These paradigms deviate from pure self-supervised audio-visual learning, heavily dependent on auxiliary modalities or pre-trained models. Furthermore, an investigation into existing multi-source benchmarks exposes a systematic evaluation bias: evaluating continuous localisation heatmaps against rectangular bounding boxes treats all enclosed pixels equally, rewarding bloated over-activation that encompasses large background corners (e.g. for diagonally oriented instruments like flutes or irregular shapes like turkeys) while severely penalising genuine, boundary-adherent spatial localisation.
Drawing inspiration from human auditory perception—which focuses first on the most salient acoustic event before shifting attention—the authors investigate neural network convergence under mixed audio signals. They uncover a key physical phenomenon: contrastive models do not attempt to represent all simultaneous sources evenly, but instead consistently exhibit "selective convergence" towards the most prominent audio-visual alignment. Core idea: embrace rather than suppress this natural learning bias by first using self-supervised contrastive learning to isolate the dominant sounding source as a spatial prior, and then leveraging complementary spatial masking with cross-attention decouplers to progressively unmask the subdominant source, converting an ill-posed joint inverse problem into a sequence of well-conditioned self-supervised optimisations.
Method¶
Overall Architecture¶
The SCAV (Selective Convergence Audio-Visual localisation) framework consists of two progressive training stages. In Stage 1, a frozen pre-trained visual encoder and a trainable audio encoder are trained under mixed audio-visual inputs using contrastive learning; gradient simplicity bias drives selective convergence, yielding a coarse localisation heatmap that isolates the dominant sound source as a spatial prior. In Stage 2, this spatial prior divides the visual features into dominant and complementary regions; a visual transformer decoupler and a cross-attention audio decoupler then refine these representations into source-specific pairs, which are trained via strictly within-sample contrastive learning. At inference time, the model executes a single forward pass without iterative cycles, achieving a real-time speed of 43.2 FPS on an NVIDIA A100 GPU.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Video Frame I + Mixed Audio Amix"] --> B["Feature Extraction & Similarity<br/>Frozen ResNet18 extracts V, Audio Encoder extracts a"]
B --> C["Stage 1: Selective Convergence Prior<br/>Differentiable threshold contrastive loss yields Mdom"]
C --> D["Stage 2: Complementary Partitioning<br/>V1 = V ⊙ Mdom, V2 = V ⊙ (1 - Mdom)"]
D --> E["Cross-Attention Decoupling<br/>Audio decoupler uses V-features to separate fa1, fa2"]
E --> F["Within-Sample Contrastive Learning<br/>Lstage2 optimisation, upsampled outputs M1, M2"]
Key Designs¶
1. Selective Convergence Discovery and Dominant Spatial Prior: Resolving Circular Deadlocks via Gradient Bias
To circumvent the lack of unmixed audio signals in dual-source scenarios, Stage 1 does not force the model to capture both sources simultaneously. Given an image \(I \in \mathbb{R}^{H \times W \times 3}\) and a mixed log-spectrogram \(A_{\text{mix}} \in \mathbb{R}^{T \times F}\), a visual encoder \(\Phi_v\) (ImageNet pre-trained ResNet-18, kept frozen) extracts feature map \(V \in \mathbb{R}^{D \times h \times w}\) while a trainable audio encoder \(\Phi_a\) extracts vector \(\mathbf{a} \in \mathbb{R}^D\). Calculating spatial dot-product similarities \(\mathcal{S}_{i \to j} = \langle V_j, \mathbf{a}_i \rangle \in \mathbb{R}^{h \times w}\), soft positive and negative masks are obtained via differentiable thresholding: $$ \hat{m}p = \sigma\left(\frac{\mathcal{S}} - \epsilon_p}{\tau}\right), \quad \hat{mn = 1 - \sigma\left(\frac{\mathcal{S}\right) $$ where } - \epsilon_n}{\tau\(\epsilon_p = 0.65, \epsilon_n = 0.4, \tau = 0.03\). Positive scores \(P_i\) and negative scores \(N_i\) (incorporating cross-sample negatives from \(j \neq i\)) are aggregated to compute the contrastive loss \(\mathcal{L}_{\text{stage1}}\). Driven by simplicity bias in gradient optimisation, the network naturally converges onto the single source with the strongest audio-visual co-occurrence, generating a focused, low-noise dominant source prior heatmap \(\mathbf{M}_{\text{dom}}\).
2. Complementary Spatial Decoupling Architecture: Progressive Top-Down Isolation of Remaining Sources
To expose the subdominant source obscured by the dominant one, Stage 2 exploits \(\mathbf{M}_{\text{dom}}\) as a spatial partition operator. Projected visual features endowed with 2D sinusoidal positional encodings are element-wise gated into two complementary views: dominant region \(V_1 = \tilde{V} \odot \mathbf{M}_{\text{dom}}\) and residual region \(V_2 = \tilde{V} \odot (1 - \mathbf{M}_{\text{dom}})\). A 4-layer Transformer visual decoupler \(\Phi_v^{\text{dec}}\) refines these maps into source-specific features \((f_{v1}, f_{v2})\). Next, the audio decoupler \(\Phi_a^{\text{dec}}\) treats the mixed audio embedding \(\tilde{\mathbf{a}}\) as Query, while taking \(f_{v1}\) and \(f_{v2}\) respectively as Key and Value in cross-attention: $$ f_{a1}, f_{a2} = \Phi_a^{\text{dec}}(\tilde{\mathbf{a}}, f_{v1}, f_{v2}) $$ By using spatial visual tokens to guide attention over the audio embedding, this cross-attention mechanism physically decomposes the mixed audio features into individual source components without requiring unmixed acoustic ground truth.
3. Within-Sample Decoupling Contrastive Objective: Eliminating Cross-Sample Negative Interference
With decoupled feature pairs \((f_{v1}, f_{a1})\) and \((f_{v2}, f_{a2})\), the learning objective undergoes a structural transition. While Stage 1 requires cross-sample negatives to build inter-video discriminability across the entire mini-batch, Stage 2 focuses entirely on the intra-sample assignment problem—identifying whether an acoustic component belongs to object 1 or object 2 inside the very same scene. Retaining cross-sample negatives at this stage dilutes the fine-grained within-frame boundary contrast. Consequently, Stage 2 excludes cross-sample negatives from \(N'_i\) in the contrastive objective: $$ \mathcal{L}_{\text{Stage2}} = \mathcal{L}_1 + \mathcal{L}_2, \quad \mathcal{L}_i = -\log \frac{\exp(P_i)}{\exp(P_i) + \exp(N'_i)} $$ This loss focuses gradients directly on separating the two co-occurring sources within the frame, driving each decoupled branch to attend exclusively to its assigned sounding entity.
Loss & Training¶
The optimisation follows a strict two-stage schedule: 1. Stage 1: Train the audio encoder \(\Phi_a\) using \(\mathcal{L}_{\text{stage1}}\) while keeping \(\Phi_v\) frozen, with a batch size of 256 and Adam optimiser at learning rate \(10^{-4}\) until selective convergence stabilises \(\mathbf{M}_{\text{dom}}\). 2. Stage 2: Fix the spatial prior generation, initialising decoders and projection layers, and optimise the entire pipeline end-to-end via \(\mathcal{L}_{\text{Stage2}}\) with the same batch size and learning rate. 3. Inference: During evaluation and deployment, the trained network runs an uninterrupted single forward pass. Independent heatmaps are produced via inner products and bilinear upsampling: \(\mathbf{M}_k = \text{Upsample}(\langle f_{vk}, f_{ak} \rangle)\), running at 43.2 FPS on a single NVIDIA A100 GPU.
Key Experimental Results¶
Main Results¶
SCAV was extensively benchmarked on MUSIC-Duet, VGGSound-Duet (bounding-box evaluation), and the newly established VGGSound-DuetMask (pixel-precise evaluation):
| Dataset | Annotation | Method | Supervision | CAP (%) | [email protected] (%) | AUC (%) |
|---|---|---|---|---|---|---|
| MUSIC-Duet | BBox | AVGN (CVPR 2023) | Weakly-sup. | 50.6 | 32.5 | 24.6 |
| MUSIC-Duet | BBox | Mix-and-Localize (CVPR 2022) | Self-sup. | 47.5 | 26.5 | 21.5 |
| MUSIC-Duet | BBox | NoPrior (CVPR 2024) | Self-sup. | 21.7 | 2.5 | 13.2 |
| MUSIC-Duet | BBox | SCAV (Ours) | Self-sup. | 47.6 | 47.1 | 28.6 |
| VGGSound-Duet | BBox | AVGN (CVPR 2023) | Weakly-sup. | 21.9 | 26.2 | 23.8 |
| VGGSound-Duet | BBox | Mix-and-Localize (CVPR 2022) | Self-sup. | 16.3 | 21.1 | 20.5 |
| VGGSound-Duet | BBox | SCAV (Ours) | Self-sup. | 53.0 | 53.3 | 32.0 |
| VGGSound-DuetMask | Mask (SAM) | AVGN (CVPR 2023) | Weakly-sup. | 33.61 | 18.44 | 17.39 |
| VGGSound-DuetMask | Mask (SAM) | LVS (CVPR 2021) | Self-sup. | 18.00 | 9.53 | 10.28 |
| VGGSound-DuetMask | Mask (SAM) | EZ-VSL (ECCV 2022) | Self-sup. | 16.82 | 9.58 | 11.35 |
| VGGSound-DuetMask | Mask (SAM) | SCAV (Ours) | Self-sup. | 39.30 | 33.59 | 23.52 |
Ablation Study¶
Component-wise ablations on VGGSound-Duet assess the impact of spatial priors, progressive scheduling, and negative sampling strategies across both bounding-box and mask benchmarks:
| Setting # | Stage 2 Decoupler | Spatial Prior | Progr. Train | Cross-neg. | Duet (BBox) [email protected] (%) | DuetMask [email protected] (%) | Note |
|---|---|---|---|---|---|---|---|
| (a) | ✓ | ✗ (Uniform) | ✓ | ✗ | 28.94 | 16.15 | Uniform mask fails to provide spatial grounding |
| (b) | ✗ (Stage 1 only) | — | — | ✗ | 32.66 | 18.84 | Stage 1 single-branch evaluation |
| (c) | ✓ | ✓ | ✗ (Joint end-to-end) | ✗ | 41.84 | 25.62 | Joint training suffers from unstable early priors |
| (d) | ✓ | ✓ | ✓ | ✓ | 26.69 | 17.40 | Adding cross-sample negatives dilutes intra-mixture contrast |
| Full Model | ✓ | ✓ | ✓ | ✗ | 53.27 | 33.59 | Complete SCAV two-stage sequential convergence |
Moreover, empirical verification of selective convergence in Stage 1 shows that the dominant source achieves 63.39% CAP, 61.70% [email protected], and 37.82% AUC, whereas the second source attains only 13.91% CAP (-49.48%), 3.62% [email protected] (-58.08%), and 8.12% AUC (-29.70%), validating that the model spontaneously converges to a single source.
Key Findings¶
- Progressive Training Prevents Early Prior Drift: Comparing Setting (c) with the full model demonstrates that two-stage progressive training yields an 11.43% [email protected] gain on BBox and 7.97% on Mask. Joint end-to-end training fails because premature, noisy heatmaps from an unconverged Stage 1 mislead the Stage 2 decoupler.
- Cross-Sample Negatives Are Harmful in Decoupling: Introducing cross-sample negatives in Setting (d) degrades [email protected] from 53.27% to 26.69%. The decoupling phase is inherently an intra-sample source assignment challenge; enforcing inter-sample separation obscures the delicate boundaries between co-occurring sources within the same scene.
- Pixel-level Mask Benchmark Rectifies Evaluation Distortions: Moving from bounding-box metrics to SAM-generated segmentation masks reveals that prior baselines (such as LVS and EZ-VSL) hover below 10% [email protected] due to background over-activation, while SCAV achieves 33.59%, proving its genuine boundary-conforming localisation capability.
Highlights & Insights¶
- Turning Learning Bias into Algorithmic Opportunity: While simplicity bias in contrastive learning is commonly considered a defect causing representation suppression, this work astutely transforms it into an emergent attention filter to dissolve the circular dependency.
- Establishing the Largest Pixel-Level Dual-Source Benchmark: By augmenting 3,951 test pairs across 220 sound classes with SAM segmentation masks, VGGSound-DuetMask resolves the long-standing evaluation distortion caused by bounding boxes and establishes a reliable foundation for future multi-source research.
- Real-Time Single-Pass Inference: Despite requiring two training stages to guarantee stable spatial priors, the final model operates in a single forward pass without recurring loops, achieving 43.2 FPS on an NVIDIA A100 GPU.
Limitations & Future Work¶
- Constrained to Dual-Source Scenarios: The complementary partitioning mechanism (\(V_1\) and \(V_2 = 1 - \mathbf{M}_{\text{dom}}\)) is tailored for two sources; handling three or more concurrent sound sources requires investigating hierarchical cascading or iterative unmasking schedules.
- Vulnerability to Severe Acoustic-Visual Overlap: When two sound sources exhibit closely overlapping frequency profiles (e.g. identical instruments playing in unison) and spatial proximity, the extracted spatial prior may suffer from boundary leakage.
- Future Directions: Exploring whether selective convergence can be extended to other multi-modal association tasks, such as video-to-speaker speech transcription alignment and complex interleaved image-text grounding.
Related Work & Insights¶
- vs Mix-and-Localize (CVPR 2022): Mix-and-Localize relies on cycle-consistent random walks over an image-audio graph to enforce disentanglement, suffering from optimization instability; SCAV leverages intrinsic contrastive convergence to achieve a huge boost on VGGSound-Duet [email protected] (from 21.1% to 53.3%).
- vs AVGN (CVPR 2023): AVGN relies on supervised video-level class labels to guide learnable tokens; SCAV operates in a strictly self-supervised setting without any category labels, yet surpasses AVGN across all metrics on large-scale dual-source benchmarks.
- vs OA-SSL (CVPR 2025): OA-SSL employs external Multimodal LLMs to generate descriptive contextual prompts for silent vs active object filtering; SCAV achieves state-of-the-art results using raw audio-visual pairs alone, demonstrating the efficiency and purity of self-supervised representation learning.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Discovers the selective convergence phenomenon in multi-source contrastive learning and devises an elegant progressive decoupling paradigm.]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive benchmarking across three datasets, rigorous isolation of dominant/subdominant sources, and creation of a 3,951-sample pixel-level benchmark.]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation grounded in cognitive science, coherent mathematical formulations, and rigorous visual-textual consistency.]
- Value: ⭐⭐⭐⭐⭐ [Provides an annotation-free solution for multi-source audio-visual inverse problems and rectifies benchmark evaluation protocols.]