Speaking While Listening: A Survey and Empirical Audit of Full-Duplex Spoken Dialogue Systems¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/MM-Speech/DuplexSurvey
Area: LLM Reasoning
Keywords: Full-Duplex Spoken Dialogue, Decision State Machine, Interaction Ontology, Architectural Hierarchy, Realization Gap
TL;DR¶
This paper introduces three unified formal frameworks—the L0–L3 architectural hierarchy, the \(T \times I \times R\) interaction ontology, and a 5-state decision FSM—to resolve the taxonomy crisis in Full-Duplex Spoken Dialogue Systems (FD-SDS), demonstrating through independent empirical audits on FDB v1.5 that a substantial "Realization Gap" exists between architectural reachability and actual learned behaviors due to training data bottlenecks.
Background & Motivation¶
Since OpenAI demonstrated GPT-4o's seamless conversational interruption capabilities in May 2024, Full-Duplex Spoken Dialogue Systems (FD-SDS) have rapidly transitioned from specialized industrial implementations to an industry-wide product baseline. Consequently, more than a dozen spoken dialogue models—including Moshi, MinMo, OmniFlatten, SyncLLM, LSLM, and FireRedChat—have emerged, all claiming "full-duplex" capabilities. However, the term has been overloaded to describe fundamentally incompatible internal mechanisms. An external modular scheduler that aborts speech playback upon VAD detection, an autoregressive parallel-token decoder, and a sidecar predictor consuming LLM hidden states have all been clustered indiscriminately under the same "full-duplex" umbrella. Existing academic surveys collapse this multifaceted design space onto simplistic single axes (such as cascaded vs. end-to-end, or engineered vs. learned), obscuring crucial distinctions that matter most to system architects and developers.
The primary tension lies in the severe ambiguity of current descriptive terminology and the fragmentation of evaluation regimes. Existing characterizations fail to specify three fundamental operational questions: where the duplex decision is executed in the model stack, which exact interaction scenarios the system is capable of handling, and how the conversational state transitions dynamically on a moment-by-moment basis. Under the hood, many systems that architecturally permit simultaneous listening and speaking exhibit catastrophic behavioral failures in realistic human interactions. For instance, when a user issues a short listener acknowledgment ("uh-huh", "right"), systems frequently misinterpret the input as a competitive barge-in and truncate their own speech, or they become distracted by non-addressed background speech from third parties. Because source technical reports rely on heterogeneous, non-comparable private evaluation suites, the community has remained unable to diagnose whether poor conversational flow stems from architectural bottlenecks, training data deficits, or evaluation artifacts.
This paper tackles these challenges by introducing a formal, orthogonal taxonomy paired with rigorous, independent empirical auditing. Instead of treating full-duplex functionality as a binary attribute, the authors construct three tightly coupled diagnostic frameworks: an L0–L3 Architectural Hierarchy that pinpoints the depth of the decision point, an orthogonal \(T \times I \times R\) Interaction Ontology that formalizes conversational time, user intent, and policy actions, and a 5-state Decision Finite State Machine (IDLE/LISTEN/SPEAK/WAIT/DUAL) featuring 11 atomic transitions. Grounded by independent evaluation under the unified Full-Duplex-Bench (FDB) v1.5 benchmark, this study connects architectural potential directly to training data distributions. Core idea: formalize full-duplex spoken dialogue systems across the L0–L3 architectural hierarchy, the three-dimensional interaction ontology, and a five-state decision machine, using independent empirical audits to uncover the Realization Gap and demonstrate that training-data cell coverage—rather than architectural topology—is the binding constraint on full-duplex behavioral competence.
Method¶
Overall Architecture¶
The methodological contribution of this work operates as a structured, three-tiered conceptual framework coupled with an empirical audit pipeline. First, the L0–L3 Architectural Hierarchy establishes where the decision between listening, speaking, waiting, and concurrent processing is located across the model stack. Second, the \(T \times I \times R\) Interaction Ontology decomposes continuous conversational time, communicative intent, and system actions into an orthogonal space, highlighting six canonical acid-test cells. Third, the Decision State Machine maps these interaction moments onto five operational states governed by eleven transition rules. Finally, an empirical audit protocol evaluates runnable open-weight systems under the Full-Duplex-Bench v1.5 suite, categorizing evidence across architectural reachability (E1), paper-reported claims (E2), and independently measured behavior (E3).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Spoken Dialogue Audio Streams"] --> B["L0–L3 Architectural Hierarchy<br/>Locates Duplex Decision Point"]
B --> C["T × I × R Interaction Ontology<br/>Orthogonal Temporal, Intent & Response Space"]
C --> D["Five-State Decision State Machine<br/>5 States & 11 Transitions Track Dynamics"]
D --> E["Realization Gap & Data Audit<br/>E1/E2/E3 Evidence on FDB v1.5 Benchmark"]
E --> F["L3 Continuous Representation Hypotheses"]
Key Designs¶
1. L0–L3 Architectural Hierarchy: Disentangling Decision Point from Execution Granularity
Prior surveys conflate where a decision is made with how fast it takes effect. The proposed hierarchy formalizes four distinct structural layers: - L0 Module-level: The duplex decision is made outside the LLM by an external scheduler (e.g., FireRedChat, FlexDuo, X-Talk, SoulX-Duplug). A pipeline of VAD, End-of-Turn (EoT) classifiers, and dialogue management modules directs a half-duplex generation backbone. - L1 Hidden-state-level: The decision is computed by an external sidecar predictor consuming the internal hidden states \(h_t\) of the LLM backbone (e.g., MinMo, Freeze-Omni, and non-FD streaming designs like Qwen2.5/3.5-Omni's Thinker-Talker). - L2 Token-level: Duplex decisions are embedded directly within the autoregressive discrete token sequence (e.g., Moshi, LSLM, OmniFlatten, SyncLLM, Covo-Audio, Fun-Audio-Chat). The token vocabulary encapsulates cross-stream coordination, such as silence tokens or synchronized chunk markers. - L3 Representation-level: A shared, continuous latent state causally updated by both user and assistant streams without explicit token boundaries. This remains an open architectural frontier with zero published implementations.
Crucially, the authors separate architectural depth from Execution Granularity (continuous, frame-level, chunk-bounded, or event-triggered). Systems residing at the exact same architectural tier can have radically divergent temporal properties: for instance, within L2, Moshi evaluates decisions at 80 ms frame intervals, SyncLLM is bounded by 160–240 ms chunk boundaries, and DuplexMamba operates over coarse 2–3 s slices.
2. T × I × R Interaction Ontology and Six Acid-Test Cells: Fine-Grained Triplet Decomposition
To replace coarse binary characterizations of turn-taking, the authors formulate conversational moments as triplets \((T_i, I_j, R_k)\) across three orthogonal axes: - Temporal Relation \(T\): Sequential clean turns (\(T_1\)), latched zero-gap handoffs (\(T_2\)), local overlapping speech (\(T_3\)), sustained concurrent monologues (\(T_4\)), and mutual silence (\(T_5\)). - User Communicative Intent \(I\): Information delivery (\(I_1\)), backchannel acknowledgments (\(I_2\)), conversational repair/clarification (\(I_3\)), floor claims (\(I_4\)), floor yields (\(I_5\)), disfluency/hesitations (\(I_6\)), and non-addressed third-party audio (\(I_7\)). - System Policy Response \(R\): Continue speaking (\(R_1\)), stop and yield floor (\(R_2\)), wait and monitor (\(R_3\)), emit backchannel (\(R_4\)), ignore/filter noise (\(R_5\)), and proactively initiate turn (\(R_6\)).
Out of 210 nominal combinations, the authors extract six critical Acid-Test Cells that stress-test system capabilities: 1. Standard Turn \((T_1, I_1, R_6)\): The cascaded half-duplex baseline. 2. Latched Zero-Gap \((T_2, I_1, R_6)\): Stresses semantic end-of-turn prediction latency. 3. Cooperative Barge-In \((T_3, I_4, R_2)\): The user claims the floor mid-utterance; requires stopping within ~200 ms. 4. Backchannel During System Speech \((T_3, I_2, R_1)\): The user utters "uh-huh"; the system must continue speaking without truncation. 5. Third-Party Noise Rejection \((T_3, I_7, R_5)\): Overlapping environmental or extraneous speech must be filtered out. 6. Hesitation with Long Silence \((T_5, I_6, R_3)\): Syntactically incomplete pauses ("I want to...") requiring the system to wait rather than interject.
3. Five-State Decision State Machine: Formalizing System Dynamics and State Transitions
The paper models full-duplex operation moment by moment via a finite state machine \(S = \{\text{IDLE}, \text{LISTEN}, \text{SPEAK}, \text{WAIT}, \text{DUAL}\}\) with 11 directed transitions: - Incorporates WAIT (the system pauses mid-turn during user hesitation) and DUAL (both streams are active with mutual integration). - Each transition is defined as \(s \xrightarrow{(T_i, I_j) / R_k} s'\), grouped into three families: - Onset Transitions (\(\tau_1: \text{IDLE} \to \text{LISTEN}\), \(\tau_2: \text{IDLE} \to \text{SPEAK}\)); - Turn-Handoff Transitions (\(\tau_3: \text{LISTEN} \to \text{SPEAK}\), \(\tau_4: \text{LISTEN} \to \text{WAIT}\), \(\tau_5: \text{WAIT} \to \text{SPEAK}\), \(\tau_6: \text{WAIT} \to \text{LISTEN}\), \(\tau_7: \text{SPEAK} \to \text{IDLE}\)); - Overlap Transitions (\(\tau_8: \text{SPEAK} \to \text{DUAL}\), followed by branching based on user intent: \(\tau_9: \text{DUAL} \to \text{LISTEN}\) on floor claims, \(\tau_{10}: \text{DUAL} \to \text{SPEAK}\) on backchannels, and \(\tau_{11}: \text{DUAL} \to \text{SPEAK}\) on third-party noise filtering).
Half-duplex and pseudo-duplex architectures inevitably collapse \(\tau_8\) unconditionally into LISTEN, fundamentally lacking the ability to resolve \(\tau_{10}\) and \(\tau_{11}\).
4. The Realization Gap and Data-Bottleneck Audit: Grounding Discrepancies in Data Distribution
The audit establishes a three-tier evidential framework: (E1) Architecturally reachable, (E2) Source-paper reported, and (E3) Independently measured under uniform protocols. By benchmarking runnable open models on Full-Duplex-Bench v1.5 across 498 paired scenarios, the authors reveal a profound Realization Gap: - Models sharing identical L2 architectural reach diverge drastically in behavior: on backchannel handling (\(T_3 \cdot I_2 \cdot R_1\)), Fun-Audio-Chat and Covo-Audio-Chat resume speech in 95.9% and 98.0% of cases, whereas Moshi resumes in only 4.1% (a ~24× performance disparity). - An audit of public Type-C conversational corpora (Fisher, CANDOR, MagicData-RAMC, Easy Turn, HumDial Track 2) reveals that public corpora total only ~5k hours and virtually omit sustained concurrent speech (\(T_4\)) and third-party rejection (\(T_3 \cdot I_7\)). Industrial proprietary datasets are 1–2 orders of magnitude larger. - This leads to the Data-Bottleneck Thesis: across L0–L2, training-data cell coverage sets the ceiling on observed full-duplex behavior, while architectural choices dictate sample efficiency and inductive bias.
Loss & Training¶
The paper synthesizes the standard four-stage curriculum that has emerged across modern L1/L2 full-duplex systems:
1. General Text Pre-training: Reusing pretrained foundational LLMs (e.g., Qwen, LLaMA) for linguistic reasoning and world knowledge.
2. Speech-to-Text Alignment (ASR): Aligning speech encoder representations into the LLM token or hidden-state space.
3. Text-to-Speech Alignment (TTS): Aligning autoregressive token heads or detokenizers to generate acoustic codec tokens.
4. Duplex Interaction Alignment: Training on time-synchronous dual-channel dialogues using automatic labeling recipes:
- Heuristic rule labeling (e.g., MinMo's turn-taking transition modeling);
- Silent speech token insertion (e.g., OmniFlatten's silent_speech_token);
- Time-anchor synchronization (e.g., SyncLLM's [S0]/[S1] chunk delimiters).
Optionally followed by reinforcement learning with turn-taking and interaction-quality reward formulations.
Key Experimental Results¶
Main Results: FDB v1.5 Independent Controlled Evaluation (E3 Level)¶
The evaluation executes the official Full-Duplex-Bench v1.5 protocol across 498 paired audio interaction scenarios: 200 interruptions, 98 backchannels, 100 talking-to-others, and 100 background-speech trials. A fixed-prompt GPT-4o judge evaluates whether the system redirects its response. Success requires responding to interrupting content (C_RESPOND) for interruptions (\(T_3, I_4, R_2\)) and maintaining the original response (C_RESUME) across backchannels and third-party distractions.
| System | Architecture Layer | Interruption Respond (%) \((T_3, I_4, R_2)\) |
Backchannel Resume (%) \((T_3, I_2, R_1)\) |
Talking-to-Others Resume (%) \((T_3, I_7, R_5)\) |
Background-Speech Resume (%) \((T_3, I_7, R_5)\) |
|---|---|---|---|---|---|
| Covo-Audio-Chat | L2 Token-level | 81.0 | 98.0 | 40.0 | 11.0 |
| Fun-Audio-Chat | L2 Token-level | 71.5 | 95.9 | 7.0 | 9.0 |
| Moshi | L2 Token-level | 61.0 | 4.1 | 14.0 | 7.0 |
| Mini-Omni2† | L2 Token-level | 36.7 | 85.6 | 56.6 | 30.6 |
(†Note: Mini-Omni2 evaluated on 493/498 scenarios due to 5 decoding failures; reported separately).
Strict Stop-Latency Benchmark (E3 Level)¶
To eliminate artifacts where systems appear fast simply because they were silent prior to interruption, the audit defines: \(\text{Strict Stop-latency} = t_{\text{speech\_stop}} - t_{\text{interruption\_onset}}\). Trials are screened for active speech (\(\ge 40\%\) active speech in the preceding 500 ms, \(\ge 60\%\) in the preceding 100 ms). Non-stops within 2 seconds are right-censored.
| System | Architecture / Mechanism | Candidate Trials | Eligible Trials | Observed Stops | Censored Trials | Stop Success Rate (%) | P50 Median Latency (ms) | P90 Latency (ms) |
|---|---|---|---|---|---|---|---|---|
| Moshi | L2 Parallel Token Stream | 200 | 61 | 59 | 2 | 96.7% | 341 | 1000 |
| Freeze-Omni | L1 Hidden-State Sidecar | 200 | 64 | 49 | 15 | 76.6% | 897 | 1708 |
| DuplexCascade | L0 Modular Reference Pipeline | 200 | 15 | 11 | 4 | 73.3% | 612 | 1263 |
Key Findings¶
- Enormous Variance in Backchannel Handling: Despite equivalent L2 architectural capabilities, systems exhibit a 24× spread in backchannel resumption rate (Covo-Audio-Chat at 98.0% vs. Moshi at 4.1%). Moshi misidentifies user listener acknowledgments as floor-claims in over 95% of cases, underscoring that concurrent token decoders do not automatically acquire turn-taking policies without targeted training data.
- Universal Breakdown on Third-Party Speech: All evaluated systems collapse when exposed to background non-addressed voices, showing resumption rates of only 7%–11% on background speech. This empirical failure directly mirrors the structural omission of \(T_3 \cdot I_7\) data in public corpora.
- Decision Layer Does Not Dictate Latency: Moshi achieves 341 ms median latency due to frame-level updates (12.5 Hz Mimi frames), whereas Freeze-Omni (L1) is constrained by chunk-boundary decision commits (~160 ms chunk steps), yielding an 897 ms median latency. L0 modular pipelines suffer from low eligibility and queue flushing delays.
Highlights & Insights¶
- Formal Mapping Between Ontology and FSM: Concretely connects the \(T \times I \times R\) interaction ontology to atomic FSM transitions (\(\tau_1\) through \(\tau_{11}\)), allowing negative results to be localized to specific missing transitions (e.g., absence of \(\tau_{10}\)) rather than broad claims of system failure.
- Formulation of the Data-Bottleneck Thesis: Demonstrates that for L0–L2 systems, performance ceilings are determined by the cell-coverage distribution of training datasets. This provides a clear roadmap for open-source efforts (e.g., synthesizing targeted \(T_3 \cdot I_7\) and \(T_4\) corpora).
- Theoretical Foundations for L3 Continuous Modeling: Outlines the architectural blueprint for next-generation L3 systems, identifying continuous-latent autoregressive models (e.g., MAR, CALM), dual-stream JEPA architectures, and latent world-model conditioning as viable paradigms to dissolve discrete token bottlenecks.
Limitations & Future Work¶
- Limited Runnable Model Coverage: Due to the proprietary, closed-source nature of commercial platforms (GPT-4o, closed MinMo variants), controlled E3 measurements were restricted to four open-source systems.
- Acoustic and Language Bias in Benchmarks: Over 95% of public test audio is restricted to English telephone and video-call acoustic profiles, lacking standardized multilingual and multi-microphone far-field test assets.
- Absence of Causal-Influence Probes: Current stop-latency metrics cannot fully distinguish substantive full-duplex interaction from apparent full-duplex (e.g., chunk-alternating models like SyncLLM that quickly halt decoding upon boundary triggers without integrating mid-turn acoustic semantics).
Related Work & Insights¶
- vs. WavChat (Ji et al., 2024): WavChat organizes systems along a cascaded vs. end-to-end trichotomy; this paper provides the orthogonal L0–L3 hierarchy, isolating decision location from execution timing.
- vs. Chen & Yu (2025): Chen & Yu classify models along an engineered vs. learned axis; this paper formulates conversational dynamics via a 5-state FSM and provides direct empirical evidence of the Realization Gap.
- vs. Full-Duplex-Bench Series (Lin et al., 2025b–2026b): While FDB pioneers overlap evaluation, this survey anchors its test scenarios to formal ontology cells and establishes strict stop-latency auditing rules that prevent silent-onset latency inflation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first unified formal taxonomy (L0–L3, \(T \times I \times R\), 5-state FSM) and eliminates pervasive definitional ambiguity in full-duplex dialogue.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Features extensive historical analysis spanning 2018–2026, paired with rigorous E3-level controlled benchmarking and comprehensive training corpus cell coverage auditing.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally rigorous and lucid; tightly weaves theoretical definitions, empirical data, and state-machine formalisms.
- Value: ⭐⭐⭐⭐⭐ Serves as a landmark survey and foundational compass for next-generation streaming spoken language models and full-duplex agent architectures.