Skip to content

Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/bellagodiva/av-ste.git
Area: Multimodal VLM
Keywords: Full-Duplex Spoken Dialogue Models, Audio-Visual Speech Enhancement, Streaming Speech Tokens, Noise-Adaptive Modulation, Moshi

TL;DR

Addressing the vulnerability of full-duplex spoken dialogue models to acoustic noise and multi-speaker interference, AV-STE introduces a causal streaming audio-visual token enhancement front-end that restores corrupted semantic speech tokens without retraining or modifying the frozen downstream speech LLM.

Background & Motivation

Full-duplex spoken dialogue systems, exemplified by Moshi, Freeze-Omni, and MinMo, represent a major paradigm shift toward natural human-agent interaction by supporting concurrent listening and speaking, which naturally enables human-like conversational dynamics such as spontaneous interruptions and backchannel feedback. However, current state-of-the-art architectures rely almost exclusively on audio-only perceptual streams to track conversational state. In practical deployment scenarios, speech signals are inevitably corrupted by diverse acoustic degradations, ranging from ambient non-speech noise (such as background music, urban sounds, and cafe babble) to competitive acoustic interference from overlapping speakers. Such corruptions severely distort the discrete speech representations extracted by speech tokenizers, causing the dialogue model to misinterpret conversational content and produce irrelevant or incoherent responses.

In face-to-face human interaction, visual cues—particularly rhythmic lip movements—serve as an indispensable complementary modality that disambiguates acoustic confusion under severe auditory corruption. While prior literature in audio-visual speech recognition (AVSR) has firmly established the advantages of visual speech perception, AVSR systems are fundamentally designed to predict text. Feeding AVSR into native speech-to-speech dialogue models would introduce a cumbersome cascaded pipeline involving speech-to-text and text-to-speech token conversion, incurring prohibitive computational latency and sacrificing the rich prosodic nuances encoded in native audio codecs. Conversely, recent full-duplex multimodal approaches like AV-Dialog attempt to inject visual grounding by fine-tuning the foundational speech dialogue model directly on paired audio-visual conversational corpora. Such joint end-to-end retraining is not only computationally expensive and data-intensive, but also risks catastrophic forgetting of the pretrained model's existing linguistic reasoning and multi-turn conversational competence.

This dilemma exposes a critical and timely research question: can visual speech cues enhance dialogue robustness against acoustic corruption without retraining the underlying speech LLM itself? The authors' insight is to decouple audio-visual robustness from dialogue reasoning by intervening strictly at the discrete token level. Specifically, modern multi-codebook speech tokenizers isolate semantic linguistic content into a dedicated primary codebook while assigning acoustic details to subsequent residual codebooks. By performing targeted restoration exclusively on the corrupted semantic token stream while preserving acoustic streams unaltered, one can salvage speech comprehension while keeping the entire downstream conversational backbone untouched. Core idea: propose AV-STE, a modular streaming audio-visual front-end that leverages a causal bounded-lookahead encoder, soft-token cross-attention, and entropy-guided noise-adaptive modulation to restore clean semantic speech tokens from noisy audio and lip video, enabling plug-and-play robust full-duplex conversation with a completely frozen speech LLM.

Method

Overall Architecture

The core architectural principle of AV-STE is "lightweight streaming restoration at the front-end, zero modification to the downstream speech LLM". The framework consumes two synchronized input streams from the user: the raw monaural noisy audio waveform and a 25 fps grayscale video sequence focused on the speaker's cropped mouth region. Simultaneously, the pre-existing speech tokenizer (Mimi from Moshi) processes the corrupted audio to produce candidate token distributions and discrete acoustic codes.

The end-to-end processing pipeline unfolds across three consecutive stages: first, a Streaming Audio-Visual Encoder converts the raw noisy waveform and synchronized lip frames into temporally aligned, noise-robust audio-visual embeddings under strict low-latency causal constraints. Second, a Noise-Adaptive Fusion module integrates these audio-visual features with the noisy tokenizer representations via soft-token cross-attention, while an uncertainty-driven gating mechanism dynamically balances the contribution between the tokenizer prior and cross-attended visual cues according to the estimated corruption severity. Finally, a linear classification head projects the fused representation into the discrete semantic vocabulary to predict clean target tokens. These enhanced semantic tokens are merged with the original, unperturbed acoustic codebook streams and streamed directly into the frozen full-duplex speech LLM (Moshi) to trigger conversational response generation in real time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    InAudio["User Corrupted Audio"] --> Tok["Speech Tokenizer (Mimi)<br/>Quantization & Feature Extraction"]
    InVideo["Synchronized Lip Video (25 fps)"] --> S1["Streaming Audio-Visual Encoder<br/>Causal Convolutions + L=4 Lookahead Attention"]
    InAudio --> S1
    Tok -->|Semantic Logits| S2["Soft-Token Cross-Attention<br/>AV Features as Q, Soft Tokens as K/V"]
    S1 --> S2
    Tok -->|Distribution Entropy| S3["Noise-Adaptive Modulation Gate<br/>Normalized Entropy Prior + Residual MLP"]
    S2 --> S3
    S1 --> S3
    S3 --> OutSem["Enhanced Semantic Token Stream"]
    Tok -->|Pass-Through Acoustic Streams| Moshi["Frozen Full-Duplex Speech LLM (Moshi)<br/>Streaming Conversational Response"]
    OutSem --> Moshi

Key Designs

1. Streaming Audio-Visual Encoder: causal bounded-lookahead adaptation of bidirectional multimodal backbones

Standard state-of-the-art audio-visual speech representations, such as AV-HuBERT, are built on bidirectional self-attention and non-causal temporal convolutions designed for offline processing. Naively imposing full causality disrupts the precise cross-modal synchronization between visual lip dynamics and acoustic phonation, degrading representation quality.

AV-STE re-engineers the AV-HuBERT Large backbone to reconcile temporal feature quality with streaming latency requirements. In the visual convolutional front-end, all temporal convolution layers are converted to causal convolutions with left-only temporal padding, ensuring that feature extraction at time \(t\) has zero dependency on future frames. Within the deep Transformer blocks, bidirectional attention is replaced with a bounded-lookahead attention mask: $$ M_{ij} = \begin{cases} 0, & j \le i + L \ -\infty, & j > i + L \end{cases} $$ where \(L\) denotes the allowable future frame lookahead. Setting \(L=4\) (which corresponds to exactly 160 ms latency at a 25 Hz frame rate) provides an optimal temporal receptive window that captures the biophysical delay between lip articulatory movements and acoustic sound generation without compromising interactive conversational flow. A strided causal convolution then downsamples the output sequence to match the target frame rate of the downstream speech tokenizer.

2. Soft-Token Cross-Attention: resolving discrete token deception and viseme ambiguity via continuous retrieval

Under low SNR conditions or severe speaker overlap, the speech tokenizer's discrete predictions (via hard argmax selection) are prone to severe misclassifications; conditioning directly on discrete token embeddings propagates corrupted linguistic priors into cross-modal fusion. Conversely, visual lip reading alone suffers from viseme ambiguity (where distinct phonemes like /p/, /b/, and /m/ share indistinguishable visual appearances), rendering pure visual features inferior to acoustic tokens under clean or mild noise conditions.

To tackle both discrete quantization errors and visual ambiguity, AV-STE performs cross-attention over continuous soft token probability distributions rather than discrete embeddings: $$ p_t = \text{softmax}\left(\frac{l_t}{\tau}\right), \quad e_t = p_t E W_{proj} $$ where \(l_t\) represents the raw logits from the noisy tokenizer, \(\tau=1.0\) is the temperature hyperparameter, \(E \in \mathbb{R}^{V \times d_e}\) is an embedding codebook trained jointly with the fusion module, and \(W_{proj}\) maps the soft embedding to the model dimension. The model then executes bounded-lookahead cross-attention treating the robust causal audio-visual features \(h_{av}^t\) as Queries, and the soft token embeddings \(e_t\) as Keys and Values: $$ \hat{c}t = \text{CrossAttn}(h_t) $$ Reversing the query-key roles degrades performance significantly: using noise-resilient audio-visual features as queries allows the model to actively retrieve and filter reliable linguistic hypotheses from the ambiguous token distribution.}^t, e_{1:t+L}, e_{1:t+L}), \quad c_t = \text{LayerNorm}(h_{av}^t + \hat{c

3. Noise-Adaptive Modulation: entropy-prioritized dynamic weighting of acoustic and visual modalities

Because acoustic corruption levels fluctuate dynamically during live conversations, the optimal weighting between acoustic tokenizer tokens and audio-visual features varies substantially. In clean speech, the acoustic tokenizer is highly accurate and should dominate; in deep acoustic noise or multi-talker overlap, the tokenizer distribution degrades toward random uncertainty and visual guidance must take precedence.

To automate this transition without requiring explicit external SNR detectors, AV-STE exploits the normalized entropy of the tokenizer's output distribution as an intrinsic reliability prior: $$ \hat{H}t = -\frac{\sum}^V p_{t,v} \log p_{t,v}}{\log V}, \quad \hat{\lambda{ent}^t = 1 - \hat{H}t $$ A sharp, low-entropy prediction yields \(\hat{\lambda}_{ent}^t \approx 1\) (high reliability), whereas a flattened, high-entropy distribution forces \(\hat{\lambda}_{ent}^t \approx 0\) (high uncertainty). To capture subtle contextual cues beyond scalar entropy, this prior is refined via a residual MLP conditioned on the concatenated token and audio-visual features: $$ \lambda_t = \sigma\left(w \hat{\lambda}([e_t \parallel h}^t + \text{MLP^t])\right) $$ Here \(w\) is a learnable scalar initialized to 1, and the final layer of the MLP is zero-initialized, ensuring the gate initially aligns with physical entropy before learning data-driven corrections. The final enhanced representation is formed via dynamic convex combination: \(\tilde{h}_t = \lambda_t e_t + (1 - \lambda_t) c_t\).

Loss & Training

AV-STE is trained end-to-end under supervised teacher-student alignment. The fused representation \(\tilde{h}_t\) is projected into the semantic vocabulary space to compute frame-wise cross-entropy loss against clean semantic target tokens extracted from uncorrupted speech using the clean Mimi tokenizer: $$ \mathcal{L} = -\sum_{t=1}^T \log p_\theta(y_t^{clean} \mid z_{noisy}, v) $$ To prevent the model from exploiting artificial noise artifacts, training adopts a progressive curriculum: 1. Initial Training: On the LRS3 dataset, utterances are sampled on-the-fly as clean speech (20%), mixed with AudioSet non-speech environmental noise (40%), or mixed with AudioSet conversational speech (40%), with SNRs uniformly sampled from \([-10\text{ dB}, 10\text{ dB}]\). 2. Same-Dataset Speaker Interference Tuning: To eliminate cross-dataset recording shortcuts, AudioSet conversational speech is replaced with 1 to 4 interfering speakers randomly drawn from the LRS3 training set and mixed dynamically at identical SNR ranges. This checkpoint serves as the master model for all subsequent evaluations.

Optimization is conducted using Adam (\(\beta_1=0.9, \beta_2=0.98\)) with a tri-stage learning rate schedule: 4,000 warm-up steps followed by 35,000 decay steps down to 5% of the peak learning rate (\(10^{-4}\)). Gradients are clipped at a maximum norm of 5.0. Training takes approximately 73 GPU hours on a single NVIDIA A6000 GPU across 6 epochs using FP16 mixed precision.

Key Experimental Results

Main Results: Multi-Condition Semantic Token Recovery and Full-Duplex Dialogue Performance

Evaluations cover in-domain LRS3 noise, AudioSet speaker interference, challenging same-dataset LRS3 multi-speaker interference (1-4 overlapping speakers), and out-of-domain Seamless Interaction.

The quantitative comparison of semantic token accuracy (%) across five SNR levels is presented below:

Target Domain Noise / Interference Type Model Architecture Clean 10 dB 5 dB 0 dB -5 dB -10 dB Average
LRS3 (In-domain) Non-speech (AudioSet) Mimi (Baseline) 100.00* 59.11 41.30 22.62 10.33 4.81 27.63
LRS3 (In-domain) Non-speech (AudioSet) Mimi + AV-STE 76.78 73.48 71.87 69.48 65.34 59.92 68.02
LRS3 (In-domain) Speaker (AudioSet) Mimi (Baseline) 100.00* 54.81 37.61 22.99 12.25 7.19 26.97
LRS3 (In-domain) Speaker (AudioSet) Mimi + AV-STE 76.78 73.17 71.76 69.20 65.82 60.52 68.09
LRS3 (Same-dataset) 1-4 Speakers (LRS3) Mimi (Baseline) 100.00* 38.43 15.83 5.26 2.22 1.31 12.61
LRS3 (Same-dataset) 1-4 Speakers (LRS3) Mimi + AV-STE 76.78 72.46 70.27 66.94 62.17 56.18 65.60
Seamless (OOD) Non-speech (AudioSet) Mimi (Baseline) 100.00* 36.80 22.49 12.35 5.93 3.77 16.27
Seamless (OOD) Non-speech (AudioSet) Mimi + AV-STE 43.79 41.89 40.12 38.11 34.42 29.28 36.76
Seamless (OOD) 1-4 Speakers (Seamless) Mimi (Baseline) 100.00* 35.21 21.13 12.14 6.92 3.62 15.80
Seamless (OOD) 1-4 Speakers (Seamless) Mimi + AV-STE 43.79 38.66 37.29 35.97 33.42 29.48 34.96

(Note: Mimi's accuracy on clean speech is 100.00% by definition as its outputs serve as the ground-truth targets.)

In full-duplex conversational evaluation using frozen Moshi, models were evaluated on Takeover Rate (TOR), response Latency (seconds), and GPT-4o Semantic Coherence (scale 1 to 5). Under same-dataset LRS3 speaker interference: - Baseline Moshi: Achieves an average GPT-4o coherence score of 1.42, dropping to 1.15 at -10 dB, effectively degenerating into off-topic speech. - PersonaPlex: Yields an average coherence score of only 1.17. - PASE (Offline Generative SE) + Moshi: Achieves 1.67 average coherence (2.15 at 10 dB but drops sharply to 1.12 at -10 dB, while being non-streaming and computationally heavy). - AV-STE + Moshi (Ours): Achieves a top-ranking average coherence score of 1.91, retaining a score of 1.70 even at extreme -10 dB interference. Crucially, turn-taking metrics remain essentially intact (TOR 0.568 vs 0.656, latency 0.660s vs 0.652s), proving that semantic recovery occurs without disrupting natural turn-taking dynamics.

Ablation Study: Modality Breakdown and Architectural Components

Ablation experiments on LRS3 evaluate semantic token accuracy (%) averaged across five SNR levels:

Model / Ablation Variant Clean Non-Speech Noise Avg Speaker Interference Avg Description / Key Takeaway
Mimi (Original Tokenizer) 100.00 27.63 26.97 Unenhanced baseline; collapses under severe noise
S-AVH (Audio only, A) 75.65 59.25 58.15 Causal streaming audio-only backbone without visual cues
S-AVH (Visual only, V) 32.41 35.13 35.27 Pure lip-reading baseline; heavily constrained by viseme ambiguity
S-AVH (Audio-Visual, A+V) 76.47 67.48 67.77 Basic multimodal fusion backbone without cross-attention
+ Hard-Token Cross-Attention (Hard-CA) 75.12 66.37 66.60 Hard argmax injection; introduces deceptive token noise
+ Soft-Token Cross-Attention (Soft-CA) 76.11 67.64 67.86 Continuous distribution retrieval across multimodal features
+ Noise Adaptive Modulation (AV-STE) 76.78 68.02 68.09 Dynamic entropy-prioritized gating yields the best overall accuracy

Key Findings

  • Visual information becomes disproportionately decisive at lower SNRs: At a mild 10 dB SNR, AV-STE improves token accuracy by ~14-18%. In contrast, at -10 dB SNR where the audio-only tokenizer collapses (accuracy drops to 4.81% for non-speech noise and 1.31% for same-dataset speakers), AV-STE maintains robust accuracies of 59.92% and 56.18%—representing a performance gain of more than 40-fold.
  • Continuous soft distributions outperform hard discrete sampling: Hard-token cross-attention (Hard-CA) causes a slight degradation compared to the basic dual-stream backbone (falling from 67.48% to 66.37%), showing that hard token selection introduces misleading confirmation bias under noise. Transitioning to soft-token cross-attention with NAM restores continuous uncertainty awareness and achieves the peak accuracy of 68.02%.
  • Optimal latency-accuracy trade-off at 160 ms (L=4): Lookahead ablation demonstrates that token accuracy increases monotonically from 57% at \(L=0\) (pure causal) to 66% at \(L=4\) (160 ms), after which accuracy gains plateau up to \(L=8\). Hence, \(L=4\) perfectly reconciles real-time conversational latency tolerances with multimodal temporal alignment.

Highlights & Insights

  • Surgical semantic decoupling vs monolithic fine-tuning: By identifying that dialogue understanding depends predominantly on semantic discrete tokens rather than low-level acoustic codec details, AV-STE intervenes exclusively on the first semantic codebook. This enables zero-shot compatibility with frozen multi-billion-parameter speech LLMs, bypassing the severe compute costs and risk of linguistic forgetting inherent in full-model multimodal fine-tuning.
  • Normalized entropy as a parameter-free noise probe: Formulating tokenizer distribution entropy as an unsupervised reliability prior provides an elegant physical anchor for dynamic modality weighting, enabling smooth cross-fading between audio and visual streams without requiring dedicated auxiliary noise classifiers.
  • Broad transferability: The combination of bounded-lookahead causal attention, continuous soft-token retrieval, and entropy-prioritized modulation provides a blueprint for making any discrete-token-based speech system (e.g., SpeechTokenizer, w2v-BERT, or discrete audio LLMs) noise-resilient.

Limitations & Future Work

  • Reliance on clean visual streams: The framework currently assumes uninterrupted, high-quality face and lip videos; robustness against visual corruptions (e.g., severe motion blur, off-axis face angles, facial occlusions, or missing video frames) remains to be investigated.
  • Domain shift on out-of-distribution clean speech: While AV-STE demonstrates strong relative gains under noisy conditions on out-of-domain Seamless Interaction (36.76% vs 16.27%), its clean-speech accuracy drops from 76.78% (on LRS3) to 43.79%, highlighting sensitivity to acoustic and visual distribution shifts.
  • Inherent representation smoothing on clean audio: Due to encoder-decoder reconstruction variance, AV-STE's predicted tokens do not reproduce native Mimi tokens with 100% fidelity on clean speech, introducing a slight representation mismatch.
  • vs AVSR (e.g., Auto-AVSR, MMS-LLaMA): AVSR systems output textual transcripts. Incorporating AVSR into speech dialogue models requires cascading speech-to-text and text-to-speech tokenizers, which introduces substantial round-trip latency and discards non-verbal acoustic cues. AV-STE operates natively and causally in the discrete speech token domain with minimal latency.
  • vs End-to-End Multimodal Spoken LLMs (e.g., AV-Dialog, MoshiVis): AV-Dialog requires joint end-to-end retraining of the underlying language model on audio-visual dialogue data. AV-STE operates as a lightweight (335M parameter) front-end, keeping the multi-billion-parameter speech LLM entirely frozen and preserving its original conversational behaviors.
  • vs Generative Speech Enhancement (e.g., PASE): Generative speech enhancement models like PASE reconstruct waveforms in an offline, non-causal manner, making them fundamentally incompatible with streaming full-duplex conversations. Furthermore, PASE suffers from severe hallucinations under heavy speaker interference, whereas AV-STE leverages visual lip grounding to isolate the target speaker even at -10 dB SNR.

Rating

  • Novelty: ⭐⭐⭐⭐ [Innovative formulation of modular audio-visual enhancement directly on discrete semantic speech tokens for full-duplex dialogue]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive benchmarking across multiple noise conditions, SNR levels, speaker interference scenarios, ablation studies, and full-duplex conversational metrics]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-structured methodology, informative architectural diagrams, and disciplined experimental reporting]
  • Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly efficient, and plug-and-play solution for deploying full-duplex spoken agents in challenging real-world acoustic environments]