C3ASD: Multi-Level Consistency-Driven Representation Learning for Robust Active Speaker Detection

Conference: ECCV 2026
Paper: ECCV Official
Code: To be released (authors pledged release upon acceptance)
Area: Audio & Speech
Keywords: active speaker detection, audio-visual consistency, representation learning, corruption robustness, knowledge distillation

TL;DR

Addressing the severe performance degradation of active speaker detection models under real-world noise and occlusion, C3ASD regularizes representation learning without requiring corrupted training data or architectural bloat via three complementary constraints—embedding-level cross-modal alignment, sequence-level intra-modal contrastive margin enlargement, and prediction-level confidence-masked distillation—achieving superior robustness with merely 1.02M parameters.

Background & Motivation

Active Speaker Detection (ASD) aims to identify whether each visible person in a video is speaking at each frame, providing an indispensable foundation for video conferencing, human-robot interaction, and multimedia retrieval. Because speaking actions naturally exhibit strong audio-visual synchrony—where vocal tract sound generation and facial lip movements stem from the exact same physical speech production process—contemporary mainstream architectures (such as TalkNet, LoCoNet, and Light-ASD) construct dual-stream networks to extract visual facial features and acoustic MFCC/spectrograms, fusing them via spatio-temporal attention or element-wise addition. These models have achieved remarkable detection accuracy on clean in-domain benchmark datasets such as AVA-ActiveSpeaker.

However, high benchmark scores under clean laboratory conditions mask severe vulnerabilities in unconstrained real-world environments. In practical deployments, inputs are inevitably corrupted by diverse physical distortions: auditory streams are heavily contaminated by ambient noise, background music, or overlapping speech (babble noise); visual streams frequently suffer from motion blur, head turning, low resolution, or partial face occlusion; even more challenging are in-the-wild situations where both modalities degrade simultaneously. Existing fusion paradigms typically treat audio and visual streams as independent feature pools optimized solely through standard end-to-end classification cross-entropy. In the absence of structural regularizations, models inevitably exploit fragile, modality-specific shortcuts, segregating representations into disjoint modality clusters. Once noise or occlusion shifts features at test time, these brittle embeddings easily cross the decision boundary, leading to catastrophic failure.

The core tension lies in the discrepancy between the intrinsic modality-invariance of the speaking state and the unconstrained, fragile geometry of learned feature representations. The core idea of this paper is: build a multi-level consistency-driven representation learning framework (C3ASD) that introduces speaking-aware cross-modal cosine alignment at the embedding level, track-aware supervised contrastive learning at the sequence level to expand cluster margins, and confidence-masked distillation at the prediction level, enforcing robust, modality-invariant representations without extra inference parameters or corrupted training data.

Method

Overall Architecture

C3ASD adopts the lightweight dual-stream Light-ASD as its foundational backbone, consisting of three main phases: feature extraction, multi-level consistency regularization, and temporal sequence classification. The network takes a sequence of \(T\) grayscale face crop frames (\(112 \times 112\) resolution) and \(4T\) corresponding 13-dimensional audio MFCC frames. The visual encoder \(f_v\) applies multi-scale 3D convolutions with factorized spatiotemporal kernels to produce visual embeddings \(\mathbf{e}_v \in \mathbb{R}^{B \times T \times 128}\), while the audio encoder \(f_a\) utilizes multi-scale 2D convolutions to yield audio embeddings \(\mathbf{e}_a \in \mathbb{R}^{B \times T \times 128}\). Before multimodal fusion, the raw embeddings are regularized by embedding-level inter-modality alignment and sequence-level intra-modality contrastive constraints. Next, element-wise addition fuses the modalities into \(\mathbf{e}_{av} = \mathbf{e}_a + \mathbf{e}_v\), which is fed into a weight-shared Bidirectional GRU (BGRU) for temporal context aggregation. Finally, dedicated heads produce speaking probabilities, supervised jointly by ground truth labels and prediction-level distillation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input: Face Crops (T frames) + Audio MFCC (4T frames)"] --> Enc["Modality Encoders: 3D Visual f_v & 2D Audio f_a"]
    Enc --> Inter["Embedding-Level Inter-Modality Consistency<br/>Speaking-aware cosine similarity maximization (L_inter)"]
    Enc --> Intra["Sequence-Level Intra-Modality Consistency<br/>Track-aware contrastive margin maximization (L_intra)"]
    Inter --> Fuse["Element-wise Feature Fusion: e_av = e_a + e_v"]
    Intra --> Fuse
    Fuse --> Temp["Temporal Context: Shared-Weight BGRU"]
    Temp --> Pred["Prediction-Level Consistency Distillation<br/>Confidence-masked multimodal-to-unimodal MSE (L_pred)"]
    Pred --> Out["Output: Frame-Level Active Speaker Probability"]

Key Designs

1. Embedding-Level Inter-Modality Consistency: Speaking-Aware Cross-Modal Alignment

To resolve the issue where audio and visual features reside in completely disjoint subspaces in baseline models, C3ASD introduces a cross-modal cosine alignment regularization prior to fusion. Since speech acoustics and lip motion originate from the same speech production event, aligning their latent directions establishes mutual representational redundancy: when one modality degrades under noise, the clean partner stabilizes the fused representation. Crucially, the alignment is enforced strictly over actively speaking frames \(\mathcal{S} = \{i \mid y_i = 1\}\):

\[\mathcal{L}_{\text{inter}} = 1 - \frac{1}{|\mathcal{S}|} \sum_{i \in \mathcal{S}} \frac{\mathbf{e}_a^{(i)} \cdot \mathbf{e}_v^{(i)}}{\|\mathbf{e}_a^{(i)}\| \|\mathbf{e}_v^{(i)}\|}\]

Excluding non-speaking frames is vital: silent frames capture only uncorrelated ambient background noise and stationary faces, and forcing them to align would inject spurious correlations. Mathematically, the gradient of \(\mathcal{L}_{\text{inter}}\) with respect to an embedding is strictly orthogonal to the embedding itself (the vector rejection component), ensuring that gradient descent purely rotates feature directions toward alignment without perturbing their norms. When a corrupted modality shifts by perturbation vector \(\boldsymbol{\delta}\), aligned representations ensure large fused norm \(\|\mathbf{e}_a + \mathbf{e}_v\|\), geometrically bounding the angular deviation to \(\arctan \frac{\|\boldsymbol{\delta}\|}{\|\mathbf{e}_a + \mathbf{e}_v\|}\) and preventing drift across the classification boundary.

2. Sequence-Level Intra-Modality Consistency: Track-Aware Cluster Margin Maximization

While cross-modal alignment brings audio and visual spaces into geometric proximity, it does not constrain the internal dispersion within each modality stream. Without wide separation between speaking and non-speaking classes, features near the decision boundary remain fragile against perturbations. C3ASD addresses this via supervised contrastive learning with an essential track-aware positive sample constraint. For \(\ell_2\)-normalized frame embeddings \(\bar{\mathbf{e}}_i\), positive pairs \(\mathcal{P}(i)\) are strictly restricted to frames belonging to the identical speaker face track \(g_i\) that share the same speaking label \(y_i\), namely \(\mathcal{P}(i) = \{j \neq i \mid y_j = y_i, g_j = g_i\}\), while \(\mathcal{N}(i)\) encompasses all other frames from the same track:

\[\mathcal{L}_{\text{intra}} = - \frac{1}{N} \sum_{i=1}^N \frac{1}{|\mathcal{P}(i)|} \sum_{j \in \mathcal{P}(i)} \log \frac{\exp(\bar{\mathbf{e}}_i \cdot \bar{\mathbf{e}}_j / \tau)}{\sum_{k \in \mathcal{N}(i)} \exp(\bar{\mathbf{e}}_i \cdot \bar{\mathbf{e}}_k / \tau)}\]

Naive contrastive formulations treat all identical-label frames across the entire mini-batch as positive pairs, which improperly pulls together different people speaking simultaneously and corrupts identity representation. Theoretical analysis demonstrates that at the zero-temperature limit, minimizing this track-aware objective directly maximizes the margin on the hypersphere \(\mathbb{S}^{d-1}\) and aligns its gradient step with the Fisher discriminant ratio (\(J_F = \|\boldsymbol{\mu}_1 - \boldsymbol{\mu}_0\|^2 / S_W\)). For the noise-vulnerable audio stream, this expands the inter-class similarity gap from 0.661 to 0.838 while cutting intra-track variance from 0.039 to 0.017, substantially improving noise tolerance.

3. Prediction-Level Consistency: Confidence-Masked Multimodal Distillation

Structuring latent feature geometry does not automatically guarantee that unimodal classification outputs will behave coherently under stress. C3ASD treats the superior fused audio-visual branch \(p_{av}\) as a teacher (with detached gradients) to supervise both unimodal audio \(p_a\) and visual \(p_v\) predictions. To avoid misleading students with ambiguous teacher predictions during early training stages, a dynamic confidence mask \(\mathcal{M} = \{i \mid \max(p_{av}^{(i)}, 1 - p_{av}^{(i)}) \geq \theta\}\) with threshold \(\theta = 0.7\) filters candidate frames, applying mean squared error (MSE) over reliable targets:

\[\mathcal{L}_{\text{pred}} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \left[ \left(p_a^{(i)} - p_{av}^{(i)}\right)^2 + \left(p_v^{(i)} - p_{av}^{(i)}\right)^2 \right]\]

The theoretical rationale for adopting MSE over standard KL divergence is fundamental: in the high-confidence regime selected by the mask where \(p_{av} \to 0\) or \(1\), the denominator \(p_{av}(1 - p_{av})\) of the KL gradient diverges to infinity, resulting in severe numerical instability; in contrast, the MSE gradient magnitude remains strictly bounded (\(\leq 2|\varepsilon|\)). Furthermore, as teacher calibration improves during training, the effective mask set \(|\mathcal{M}|\) naturally expands, establishing an implicit curriculum learning schedule that guides unimodal branches to suppress conflicting signals under extreme test-time corruptions.

Loss & Training

The overall learning objective balances the baseline supervision with the three consistency regularizers:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{base}} + \lambda_1 \mathcal{L}_{\text{inter}} + \lambda_2 \mathcal{L}_{\text{intra}}^a + \lambda_3 \mathcal{L}_{\text{intra}}^v + \lambda_4 \mathcal{L}_{\text{pred}}\]

where \(\mathcal{L}_{\text{base}} = \mathcal{L}_{\text{AV}} + \mathcal{L}_{\text{V}}\) denotes standard binary cross-entropy on audio-visual and visual predictions. Loss weight hyperparameters are configured as \(\lambda_1 = \lambda_4 = 0.01\) and \(\lambda_2 = \lambda_3 = 0.001\), with contrastive temperature \(\tau = 0.07\) and confidence threshold \(\theta = 0.7\). Classification logits are scaled via a temperature annealing schedule \(r = 1.3 - 0.02 \times (e - 1)\) across epoch \(e\). The model is trained on a single NVIDIA A6000 GPU for 30 epochs using the Adam optimizer with an initial learning rate of \(10^{-3}\) and an exponential decay factor \(\gamma = 0.95\). Outside of a single linear layer for audio prediction during training, the framework introduces zero additional parameters during inference.

Key Experimental Results

Main Results

On the AVA-ActiveSpeaker benchmark and the unconstrained WASD wild evaluation benchmark, C3ASD demonstrates competitive efficiency and superior accuracy compared to representative end-to-end methods trained from scratch.

Method	Pretrained Backbone	End-to-End	Params (M)	FLOPs (G)	AVA-mAP (%)	WASD-mAP (%)
TalkNet (ACMMM 2021)	✗	✓	15.7	1.5	92.3	78.4
Sync-TalkNet (MLSP 2022)	✗	✓	15.7	1.5	89.8	-
ASD-Transformer (ICASSP 2022)	✗	✓	>13.9	>1.5	93.0	-
ADENet (TMM 2022)	✗	✓	33.2	22.7	93.2	85.6
Light-ASD (CVPR 2023)	✗	✓	1.02	0.62	93.6	85.3
LR-ASD (IJCV 2025)	✗	✓	0.84	0.51	94.5	-
C3ASD (Ours)	✗	✓	1.02	0.62	93.8	86.1

Ablation Study

On the AVA-ActiveSpeaker validation set, isolating each consistency loss demonstrates their complementary and reinforcing nature (benchmarked against the reproduced Light-ASD baseline):

Config	\(\mathcal{L}_{\text{inter}}\)	\(\mathcal{L}_{\text{intra}}\)	\(\mathcal{L}_{\text{pred}}\)	AVA Val mAP (%)	Gain / Observations
Baseline	✗	✗	✗	93.61	Standard cross-entropy baseline
+ Inter-Modality	✓	✗	✗	93.70	+0.09% (largest individual gain; directly aligns spaces)
+ Intra-Modality	✗	✓	✗	93.62	+0.01% (marginal without cross-modal anchoring)
+ Prediction Distill	✗	✗	✓	93.68	+0.07% (stabilizes unimodal predictions via distillation)
Full C3ASD	✓	✓	✓	93.80	+0.19% (all three mutually reinforce representation geometry)

Key Findings

• Robustness Under Visual Occlusion: Under severe visual degradation combining random COCO object occlusions and Gaussian noise (Object Occ. + Noise), baseline Light-ASD drops to 76.86% mAP, whereas C3ASD maintains 78.90% mAP (+2.04% gain), decisively outperforming much larger models like TalkNet (70.73%) and ADENet (66.36%). Inter-modality alignment enables the clean audio stream to compensate for missing visual features.
• Resilience Under Audio-Visual Joint Corruptions: In the most grueling joint corruption scenarios, C3ASD achieves a +1.23% average mAP gain under MUSAN noise with object occlusion, and a +1.0% average gain under DEMAND noise with object occlusion (71.5% \(\to\) 72.5%). At the lowest SNR level of \(-10\) dB, C3ASD outperforms Light-ASD by +0.9% in Babble noise (65.7% \(\to\) 66.6%) and +1.5% in Natural noise (65.1% \(\to\) 66.6%).
• Geometric Transformation of Embedding Space: In shared 2D PCA projections, baseline audio and visual embeddings occupy completely disjoint clusters with a mean paired distance of 0.67; C3ASD collapses this paired distance to 0.13 (an 80.6% reduction), while average cross-modal cosine similarity increases from 0.555 to 0.834, proving that the losses genuinely reshape representation geometry.
• Cross-Backbone Generalizability: Equipping heavier backbones like TalkNet and ADENet with C3 regularization without changing their architectures yields extensive robustness gains, e.g., TalkNet mAP under MUSAN + Object Occlusion surges from 62.41% to 68.29% (+5.88%).

Highlights & Insights

• Speaking-Only Alignment Design: Filtering out non-speaking frames prevents the network from learning harmful spurious correlations between stationary facial expressions and environmental silence/noise, delivering clean supervision without extra data labels.
• Track-Aware Supervised Contrastive Filtering: By restricting positive samples to the same individual's temporal track, the model avoids conflating distinct speakers' acoustic traits while mathematically guaranteeing Fisher discriminant margin enlargement on the unit hypersphere.
• Numerically Stable Distillation Objective: Identifying the gradient divergence risk of KL divergence in saturated confidence regions led to the elegant adoption of bounded MSE paired with confidence thresholding, establishing an effective curriculum learning dynamic.
• High Gain with Zero Inference Overhead: C3ASD achieves superior corruption robustness across acoustic noise and facial occlusion without synthesizing corrupted data during training or adding a single parameter or FLOP to the runtime inference pipeline.

Limitations & Future Work

• Lack of Explicit Inter-Speaker Context Modeling: As C3ASD builds upon a single-candidate dual-stream backbone, it does not explicitly incorporate graph relational networks (such as SPELL or LoCoNet) to resolve multi-party conversation turn-taking in crowded scenes.
• Theoretical Bounds Under Extreme Joint Degradation: When audio SNR plummets below \(-10\) dB while facial regions are fully and continuously occluded, both modalities lose nearly all semantic information, where representation geometry alone cannot fully restore active speaker transitions.
• Future Directions: Extending multi-level consistency regularization to unified end-to-end pipelines that integrate face detection, tracking, and speaker verification, or exploring lightweight state-space models (e.g., Mamba) for efficient long-term temporal modeling.

Related Work & Insights

• vs TalkNet / ADENet / LoCoNet: Prior state-of-the-art models primarily rely on architectural scaling (ranging from 15M to 34M parameters) and complex spatio-temporal attention blocks, but remain susceptible to real-world corruption due to standard classification loss training; C3ASD operates on a compact 1.02M parameter model and establishes superior robustness via three complementary representation-level regularizers.
• vs CAV2Vec / Noise-Robust ASD: Existing robust audio-visual methods typically rely on multi-stage speech separation pre-tasks or synthetic noise augmentation during training; C3ASD requires neither auxiliary separation networks nor corrupted training samples, relying entirely on intrinsic consistency constraints.

Rating

• Novelty: ⭐⭐⭐⭐ [The multi-level consistency formulation is theoretically sound and mathematically derived, with crucial design refinements like speaking-frame filtering and track-aware positive sets]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensively verified across MUSAN, DEMAND, COCO occlusion, pixelation, clean benchmarks, wild benchmarks, and multiple backbone architectures]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural organization, rigorous mathematical derivations, and seamless mapping between theoretical intuitions and empirical findings]
• Value: ⭐⭐⭐⭐⭐ [High practical impact for real-world active speaker deployment on edge and low-power devices due to zero inference overhead and training-only regularization]