Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation¶

Conference: ECCV2026
arXiv: 2512.20117
Code: TBD
Area: Semantic Segmentation
Keywords: Audio-guided segmentation, multi-source disentanglement, cross-modal alignment, contrastive learning, prototype memory bank

TL;DR¶

DDAVS disentangles multi-source audio semantics using learnable queries anchored by a prototype memory bank, and introduces delayed bidirectional cross-attention in deep layers to align audio-visual modalities, comprehensively outperforming previous state-of-the-art (SOTA) methods on AVSBench and VPO multi-source/multi-category/multi-instance scenarios.

Background & Motivation¶

The task of Audio-Visual Segmentation (AVS) aims to segment sounding objects with pixel-level accuracy given audio signals and image/video frames. Unlike traditional visual segmentation which targets all visible objects, AVS only segments objects that are actually making sounds, thereby naturally requiring models to establish fine-grained cross-modal correspondences. Early methods typically followed a paradigm where audio signals are first disentangled into several semantic components via learnable queries, and then these audio semantics are used to guide visual features for segmentation in a unidirectional manner. However, in real-world scenarios, this approach exhibits two major bottlenecks.

The first bottleneck is multi-source entanglement. Real-world scenes often contain multiple sounding sources simultaneously—for instance, an engine roaring next to a person speaking, or a guitar, saxophone, and vocals mixed together in a multi-instrument performance. When existing methods disentangle audio using learnable queries, the queries are spontaneously generated in a completely self-organized latent space without explicit structural constraints. As a result, acoustically or visually more salient objects (e.g., louder or larger ones) tend to dominate the segmentation results, while weaker sound sources are suppressed or even lost. The second bottleneck is audio-visual misalignment. In some cases, the sounding objects are very small or distant in the visual frame, offering weak localization cues; in more extreme cases, the sound source is off-screen, completely lacking corresponding visual anchors. Existing unidirectional alignment strategies allow audio queries to affect visual features, but visual information cannot be fed back to the audio component to suppress irrelevant noise or reinforce scene-consistent acoustic signals. This makes the model prone to false activations when facing weak visual signals.

The core insight of this paper is that these two bottlenecks share the same underlying cause—"timely alignment" in the decoding stage actually harms high-level semantic understanding. If alignment occurs in shallow layers, low-level noise from both audio and visual modalities will interfere with each other, blurring the true semantic correspondences. DDAVS avoids this by separating disentanglement and alignment: it first utilizes an external prototype memory bank to provide a structured semantic anchoring space for audio queries (disentangling clean semantic components), and then delays the cross-modal alignment to the deep layers of the encoder for bidirectional interaction. Core Idea: Deconstruct AVS into a two-stage process of "prototype-anchored audio semantic disentanglement \(\rightarrow\) delayed bidirectional cross-modal alignment", addressing multi-source entanglement and audio-visual misalignment through a structured semantic space and deep-layer interaction, respectively.

Method¶

Overall Architecture¶

The input to DDAVS is a raw audio waveform and corresponding video frames, and the output is a pixel-level segmentation mask of the sounding regions. The entire pipeline consists of three pathways: the audio branch extracts audio features via an HTSAT encoder, which then enter the Audio Query Module (AQM) to generate a set of disentangled audio semantic query vectors; these queries are enhanced by the Contrastive Optimization Module (COM, enabled only during training) to improve discriminative robustness; the visual branch extracts multi-scale visual features via a MiT-B5 encoder; then, in the Audio-Visual Alignment Module (AVAM), the audio queries and visual features undergo delayed bidirectional cross-attention (applied only to the deep layers, i.e., stages 3-4) to align and fuse progressively, and are finally output as a segmentation mask by a lightweight decoder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Audio + Video Frames"] --> B["Audio: HTSAT Encoder"]
    A --> C["Visual: MiT-B5 Encoder"]
    B --> D["AQM: Learnable Queries +<br/>Prototype Memory Bank Cross-Attention"]
    D --> E["COM: Contrastive Learning<br/>(Enhanced Discriminative Ability during Training)"]
    E --> F["AVAM: Delayed Bidirectional Cross-Attention<br/>(Deep Stages 3-4 Only)"]
    C --> F
    F --> G["Lightweight Decoder → Segmentation Mask"]

Key Designs¶

1. Audio Query Module (AQM): Providing a Structured Semantic Anchor for Disentangled Queries with a Prototype Memory Bank

The key challenge that AQM aims to solve is that allowing learnable queries to spontaneously generate semantic components directly from audio features results in uncontrollable "meanings" for each vector—some queries might collapse into the same sound source, while the semantic space learned by others might not assist downstream segmentation. DDAVS addresses this by introducing a global prototype memory bank to provide prior knowledge of categories.

Specifically, this is achieved in three steps. First, memory bank construction: audio clips containing only a single sounding source for each category (e.g., only guitar or only human voice) are selected from the training set; HTSAT is used to extract features, and K-means++ clustering is applied to the features of each category, fixing several features near each cluster center as the prototype vectors of that category. The prototypes of all categories are concatenated into a global memory bank. This memory bank is frozen during both training and inference, ensuring a globally consistent semantic baseline. Second, query generation: Q-Former is used to map the audio token sequence output by HTSAT into a set of learnable queries (default is 5), each naturally focusing on different audio components. Third, bank-guided refinement: this initial set of queries interacts with the prototype memory bank via cross-attention—using queries as Q and the prototypes in the bank as K/V—to retrieve the most relevant semantic prototype information to calibrate their own semantic directions. This produces disentangled semantic queries that preserve query diversity while possessing prior category knowledge and structured spatial constraints.

2. Contrastive Optimization Module (COM): Enhancing Discriminative Robustness of Queries via Augmented Contrastive Learning

Although AQM provides queries with structured semantic anchors, in multi-source overlapping scenarios, weaker audio sources can still be overshadowed by dominant ones, leading to insufficient semantic separation among different queries. The core idea of COM is to pass the same audio segment through two data augmentation paths (original audio and noise-injected pitch-shifted audio), pulling together the positive pairs of the same query under different augmentations and pushing apart negative pairs between different queries. This allows each query to stably bind to its source semantics without being disturbed by various acoustic perturbations.

Specifically, the raw audio undergoes a chain of augmentations (reverberation, pitch shifting, dynamic range compression, volume jittering) to obtain an augmented version, but this augmented version is only used in the COM branch and does not affect the audio features of the main segmentation path. Then, the original query set and the augmented query set are fed into projection heads for L2 normalization, after which an InfoNCE contrastive loss is applied between them—forcing positive pairs of the same query to be close and negative pairs of different queries to be far apart. This contrastive learning is only applied during training, introducing no extra computational overhead during inference, but it significantly improves the discriminability of queries for weak sources (adding AQM+COM yields a roughly 2.2% J&F improvement over AQM alone in the ablation study).

3. Audio-Visual Alignment Module (AVAM): Delayed Bidirectional Cross-Attention Alignment

Alignment in traditional AVS methods is either unidirectional (audio \(\rightarrow\) visual) or bidirectional at every layer with a gating mechanism that only scales the audio intensity. AVAM in DDAVS makes design choices in two aspects. The first choice is "delayed alignment": cross-modal cross-attention is applied only to stages 3-4 (deep layers) of the 4-stage Transformer block, skipping the low-level features in the first two stages. The motivation is that shallow features primarily encode low-level information like edges and textures; if audio information is injected at this stage, noise will interfere with each other. In deep layers, visual features already possess instance-level semantic structures (such as a complete human contour rather than fragmented edges), and the audio semantics are also clearly disentangled. Performing alignment at this stage establishes precise "sounding object \(\rightarrow\) acoustic category" correspondences. Experiments also verified that aligning in the first two stages harms performance (aligning stage 1 alone performs worse than the baseline without it).

The second choice is "bidirectional": each round of alignment is not a unidirectional process of audio queries attending to visual tokens. Instead, the audio queries first attend to visual tokens to extract sounding-related regions, and then the updated audio embeddings are used as K/V to attend to the visual stream, injecting discriminative acoustic cues back into the visual representations. The audio \(\rightarrow\) visual attention serves a denoising and focusing function (using the stable spatial structure of the visual modality to constrain the relatively noisy audio representations), while the visual \(\rightarrow\) audio attention uses semantically rich audio features to sharpen the visual activation maps. This process is repeated and progressively refined at each alignment depth.

Loss & Training¶

The total loss consists of four terms: cross-entropy loss for pixel-level supervision, Dice loss and IoU loss to encourage region completeness and boundary alignment, and the contrastive learning loss of COM. The weights of the four losses are regulated by hyperparameters \(\lambda\). The visual backbone is MiT-B5, and the audio encoder is HTSAT (pretrained on AudioSet). Training is conducted on 8\(\times\)RTX 4090 GPUs using the AdamW optimizer with an initial learning rate of 1e-4 and a batch size of 64.

Key Experimental Results¶

Main Results¶

Dataset	Metric	DDAVS	Prev. SOTA	Gain
AVS-Objects-S4 (Single-source)	J&F	92.4	DDESeg: 91.1	+1.3
AVS-Objects-MS3 (Multi-source)	J&F	76.0	CCFormer: 75.6	+0.4
AVS-Semantic (Semantic)	J&F	52.9	AAVS: 50.9	+2.0
VPO-MS (Multi-source)	J&F	76.1	DDESeg: 74.3	+1.8
VPO-MSMI (Multi-source Multi-instance)	J&F	72.8	RAVS: 69.3	+3.5

The improvement is most significant on the most challenging AVS-Semantic subset (which involves spatial and category ambiguities) and VPO-MSMI (multi-source multi-instance), outperforming the previous SOTA by 2.0% and 3.5% J&F, respectively. This demonstrates that disentangled semantics combined with delayed bidirectional alignment offers an obvious advantage in handling complex scenarios.

Ablation Study¶

Configuration	AVS-MS3 J&F	AVSS J&F	Description
Baseline (w/o AQM/COM/AVAM)	69.7	48.6	Encoder + Transformer + Decoder only
+AQM	71.9	49.8	Contribution of prototype-bank query disentanglement: +2.2/+1.2
+AQM+COM	74.1	51.7	Significant contribution of contrastive learning: +4.4/+3.1
+AVAM	73.2	51.5	Contribution of delayed bidirectional alignment: +3.5/+2.9
+AQM+AVAM	73.8	51.5	Combination of disentanglement + alignment
DDAVS (All)	76.0	52.9	Synergy of all components, outperforming baseline by +6.3/+4.3

Key Findings¶

COM is the largest contributor: Under the condition that AVAM is already in place, adding AQM+COM can boost J&F from 73.2 to 76.0, proving that optimizing query separability is an independent and vital dimension even after spatial alignment is established.
Delaying alignment to deep layers (Stages 3-4) is the optimal configuration: Performing cross-attention in stages 1-2 even degrades performance below the baseline; aligning only in stage 4 is inferior to stages 3+4; extending to more stages (such as 1-4) drops points due to the introduction of low-level noise.
Even if the prototype memory bank is completely removed (OOB-100%), the model still outperforms the baseline, demonstrating that the structured spatial constraints provided by prototype anchoring are effective but the model does not fully depend on them, showcasing strong robustness.
t-SNE visualization shows that the audio representations of DDAVS do not collapse to a single category in multi-source scenarios. Instead, they form interpolated distributions among the components of the mixed sources, and maintain category consistency across datasets (AVSBench vs. ESC-50).

Highlights & Insights¶

Elevating "delayed alignment" from an engineering intuition to a systematic design principle: By performing bidirectional cross-attention in the deep layers instead of aligning at every layer, the model concisely avoids interference from low-level noise on cross-modal semantic mapping. This perspective of "the timing of alignment is more important than the method of alignment" is also inspiring for other multimodal tasks (e.g., video captioning, visual question answering).
The prototype memory bank is neither a fine-tuning mechanism nor an external classifier: It remains frozen and only provides semantic anchors for queries via cross-attention—guiding rather than dominating during training, avoiding overfitting while providing sufficient structural guidance.
Augmented contrastive learning only affects query discriminability without altering the raw audio features: The augmented version only passes through the COM branch, and the main segmentation path entirely operates on clean, raw audio, which subtly decouples the noise introduced by augmentation from the segmentation learning of the main task.

Limitations & Future Work¶

The current validation is restricted to benchmark scenarios with well-defined misalignment ranges and manual classifications; open-domain videos and streaming audio scenarios have not yet been covered.
The construction of the prototype memory bank relies on the selection of single-source clips in the training set and K-means clustering. If the quality of the training set is low or the category coverage is insufficient, the effectiveness of the bank as a global anchor will degrade.
The total number of model parameters is about 150M with FLOPs around 86G (2.16G higher than the baseline), reflecting moderate-to-high computational requirements.

vs. DDESeg (CVPR 2025): Both use a feature bank to assist audio disentanglement. However, DDESeg approximates semantics via KNN from the bank (limited to the semantic space of K similar categories), while DDAVS uses learnable queries and cross-attention to flexibly retrieve semantic combinations from the bank on demand. Furthermore, the gated bidirectional alignment in DDESeg only scales audio intensity, whereas the full cross-attention in DDAVS allows for a more thorough, symmetric bidirectional interaction.
vs. AVESFormer: Although it also proposes delayed alignment, the alignment in AVESFormer is unidirectional (audio \(\rightarrow\) visual) without a visual-to-audio feedback loop, making it difficult to handle misalignment scenarios such as off-screen sources.
vs. Contrastive AVS methods (CAVP, DiffusionAVS): These methods apply contrastive learning to audio-visual modal alignment (pulling paired audio-visual features closer and pushing unpaired ones apart), whereas DDAVS applies it to query discrimination within the audio modality (not involving cross-modal learning), which aims at a completely different objective.

Rating¶

Novelty: ⭐⭐⭐⭐ It systematically integrates three relatively independent designs—delayed alignment, prototype-bank anchored disentanglement, and contrastive learning-based discriminative enhancement—into a two-stage framework, which is concise and effective. However, each individual component does not represent a completely unseen paradigm.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ The method is compared against 20+ baselines on 5 subsets across 2 benchmarks, with exhaustive ablation studies (varying components, locations, query numbers, bank robustness, and backbone combinations). The t-SNE visualization also complements the semantic analysis, making it highly solid.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation is clear. The introduction provides three sub-figures to clearly contrast the differences among the three types of methods. The experiments are well-organized, and the captions and tables are highly legible.
Value: ⭐⭐⭐⭐ AVS is a practical and challenging task. DDAVS achieves consistent improvements across multiple difficult scenarios. The method is clean, and the design principles are insightful, although it primarily constitutes an incremental improvement.