Skip to content

SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/holi-lab/SpokenUS
Area: Dialogue Systems
Keywords: Task-Oriented Dialogue, User Simulator, Spoken Dialogue, Turn-Taking, Barge-in

TL;DR

To address the lack of realistic spoken user behaviors in text-based user simulators and the lack of proactive turn-taking in omni models, this paper introduces SpokenTODโ€”a 1,034-hour spoken TOD dataset augmented with four authentic speech behaviorsโ€”and SpokenUS, a grounded spoken user simulator featuring a dedicated turn-taking head for real-time proactive barge-in decisions.

Background & Motivation

Building robust voice assistants requires training and testing them against the full complexity and messiness of real human speech interactions. Real conversations rarely follow scripted turn sequences; they naturally exhibit cross-turn slot deliveries (chunking long sequences and self-repairing along the way), frequent disfluencies (filled pauses, false starts, repetitions), emotional variations, and mid-utterance interruptions (barge-ins). User simulators (US) provide a cost-effective alternative to expensive human trials for developing and evaluating task-oriented dialogue (TOD) systems. However, modern LLM-based user simulators operate almost exclusively on text. Even when paired with a cascaded Text-to-Speech (TTS) module, they remain fundamentally passive: they can only speak after receiving a completed assistant utterance, leaving the system's ability to handle proactive interruptions completely unmodeled and untested.

Recent end-to-end omni-modal and full-duplex speech models offer a native speech-to-speech alternative, but they introduce other fundamental limitations. First, omni models lack architectural mechanisms for proactive turn-taking decisions, preventing them from intentionally interrupting agents at contextually appropriate moments. Second, because these models are pre-trained and instruction-tuned as helpful, compliant assistants, prompting them into a user role causes severe role driftโ€”often slipping into help-providing rather than help-seeking behaviors. Furthermore, they tend to dump all goal constraints in the initial turns (front-loading), fundamentally diverging from human cognitive patterns of incremental information disclosure. Underpinning all these challenges is severe data scarcity: existing spoken TOD datasets are severely limited in scale and domain coverage, with no systematic pipeline available to augment text-based TOD corpora with rich spoken phenomena.

This paper addresses these gaps by establishing a systematic spoken user behavior augmentation pipeline over rich text TOD corpora and coupling streaming listening with proactive turn-taking. Core idea: by constructing SpokenTOD, a 1,034-hour spoken dialogue dataset augmented with cross-turn slots, barge-ins, disfluencies, and emotions, and training SpokenUS with a dedicated turn-taking head and decoupled transcript-to-speech generation, the model dynamically determines interruption timing and gradually discloses goal slots while presenting a rigorous testbed for voice agent robustness.

Method

Overall Architecture

SpokenUS aims to pursue a multi-turn user goal while exhibiting realistic spoken conversational dynamics. Its inputs include the user goal and demographic profile, conversation history as interleaved text and speech tokens, and the assistant's speech delivered as a real-time token stream. The simulator operates across three sequential modes: Listening Mode, where a dedicated turn-taking head monitors streaming assistant speech tokens to decide when to yield or cut in; Pre-scripting Mode, which autoregressively plans an explicit transcript containing structured emotion tags and disfluency tokens; and Speaking Mode, which autoregressively generates discrete speech tokens conditioned on the transcript, decoded into waveform audio via conditional flow matching and a neural vocoder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Goal, Profile & Dialogue History"] --> B["Stage 1: Streaming Listening & Turn-Taking<br/>Linear-recency weighted monitoring of assistant tokens"]
    B -->|Detects Barge-in or Turn-end| C["Stage 2: Explicit Transcript Pre-scripting<br/>Generates structured text with emotion & disfluency tokens"]
    C --> D["Stage 3: Discrete Speech Synthesis & Rendering<br/>CFM flow matching and HiFi-GAN vocoder decoding"]
    D --> E["Output: High-fidelity Spoken TOD Speech"]

Key Designs

1. Spoken TOD Data Augmentation Pipeline: Grounding four spoken user behaviors on rich text corpora To overcome the bottleneck of collecting spoken TOD data from scratch, the paper introduces a dataset-agnostic augmentation framework applied across four benchmark datasets (ABCD, EmoWOZ, SGD, Taskmaster). First, for Cross-Turn Slots, alphanumeric sequences (e.g., phone numbers, emails) are segmented into natural cognitive chunks, and self-corrections are injected into 20% of them (\(P_{\text{error}}=0.20\)). Second, for Barge-in, seven combinations across three conversational functions (Error Recovery, Clarification, Efficiency) and three response styles (Implicit, Raw, Interpreted) are injected into 25% of user turns via Qwen3-32B, truncating the assistant utterance at natural trigger points. Third, for Disfluency, an exponential length-dependent model \(P(\text{disfluent} \mid L) = 1 - b^L\) (\(b=0.9453\)) determines disfluent turns, concentrating six disfluency typesโ€”filled pauses ([FP]), discourse markers ([DM]), editing terms ([EDIT]), repetitions ([REP]), corrections ([COR]), and restarts ([RST])โ€”near slot entities. Finally, for Emotion, turns are labeled with seven OCC-grounded affective states and synthesized via Qwen3-TTS with a pool of 542 diverse reference speakers, yielding SpokenTOD (52,390 dialogues, 1,034 hours of audio).

2. Dedicated Turn-Taking Head with Linear-Recency Weighting: Microsecond interruption and turn-yielding decisions To support active floor management without awaiting full audio files, SpokenUS attaches a dedicated Turn-Taking Head atop the backbone's final hidden states during Listening Mode. Assistant audio arrives as a 25 Hz stream of discrete speech tokens. To eliminate spurious edge triggers from momentary acoustic fluctuations, the head aggregates token-level logits over a historical window using a linear-recency weighting scheme: $\(P_{\text{act}} = \sum_{i=1}^W w_i \cdot \text{Softmax}(H_{\text{assistant}}^{(i)} W_{\text{turn}})\)$ The head discriminates across three action classes: listen (maintain floor yielding), barge-in (immediate interruption), and turn-end (assistant completion). Supervision assigns listen to all assistant tokens except the trailing 6 tokens (~240 ms, aligning with human neurological reaction latency), which are labeled as turn-end or barge-in. Detecting either non-listen label immediately halts assistant audio ingestion and triggers response formulation.

3. Explicit Transcript-Guided Two-Stage Speech Generation: Balancing transparency and acoustic expressiveness End-to-end black-box speech models frequently drop critical slot values in high-entropy acoustic latent spaces. SpokenUS explicitly decouples language planning from acoustic rendering. Upon turn-taking activation, the text head autoregressively produces an explicit transcript enclosed by <BOT> and <EOT>. The transcript initiates with a normalized emotion token (e.g., <|dissatisfied|>) and weaves linguistically motivated disfluency markers directly into the text stream. Subsequently, starting from <BOA>, the speech head autoregressively generates discrete speech tokens (using CosyVoice3's 6,561 finite scalar quantized tokens) conditioned on the explicit transcript until <EOA>. The resulting tokens are fed into a 300M parameter Conditional Flow Matching (CFM) diffusion module and a HiFi-GAN vocoder conditioned on reference speaker embeddings, ensuring authentic acoustic realization of hesitation and emotional prosody.

Loss & Training

SpokenUS is initialized from Qwen2.5-3B and fine-tuned in two stages. Stage 1 trains on text-only SpokenTOD dialogues for 3 epochs to instill task-oriented domain knowledge, slot tracking, and dialogue policy execution. Stage 2 jointly optimizes all three heads on user turns for 12,000 steps using a multi-task objective: $\(\mathcal{L} = \mathcal{L}_{\text{text}} + \mathcal{L}_{\text{speech}} + \mathcal{L}_{\text{turn-taking}}\)$ where \(\mathcal{L}_{\text{text}}\) and \(\mathcal{L}_{\text{speech}}\) are cross-entropy losses over text and discrete speech token generation, and \(\mathcal{L}_{\text{turn-taking}}\) supervises the 3-class turn-taking decisions. The downstream CFM and HiFi-GAN modules are fine-tuned on SpokenTOD user speech for 30,000 steps on 5ร— H200 GPUs.

Key Experimental Results

Main Results

The primary benchmark evaluates user simulators interacting with a fixed cascaded voice assistant (Whisper-large-v3 + GPT-4.1-mini + Qwen3-TTS) across 100 held-out SpokenWOZ test goals. Metrics include Goal Alignment (GA), Slot Match Rate (SMR), Human Mean Opinion Score (MOS, covering Naturalness N, Conversational Flow C, and Spoken User Behavior S), Word Error Rate (WER) measuring text-speech alignment, and WavLM speaker embedding cosine similarity (\(\text{SIM}\)).

Model Audio (hrs) GA โ†‘ SMR โ†‘ MOS (Avg) โ†‘ WER โ†“ \(\text{SIM}_{0,i}\) \(\text{SIM}_{i-1,i}\)
Human Reference โ€” 1.00 1.00 3.67 โ€” 0.84ยฑ.07 0.88ยฑ.04
Larger Omni Models
Qwen2.5-Omni-7B โˆผ3M 0.80 0.97 3.03 10.53 0.97ยฑ.03 0.96ยฑ.04
GLM-4-Voice-9B โˆผ700K 0.33 0.79 2.32 15.33 0.94ยฑ.04 0.96ยฑ.03
Qwen3-Omni-30B-A3B โˆผ20M 0.93 0.99 3.18 15.05 0.95ยฑ.03 0.95ยฑ.04
Comparable Models
Qwen2.5-Omni-3B โˆผ3M 0.36 0.93 2.34 15.09 0.90ยฑ.04 0.92ยฑ.03
InteractiveOmni-4B โˆผ465K 0.59 0.89 2.63 15.68 0.95ยฑ.03 0.94ยฑ.04
SpokenUS (3B, Ours) โˆผ1K 0.82 0.99 4.06 11.36 0.93ยฑ.02 0.92ยฑ.03

Stress-Testing Voice Agents & Interruption Detection

To evaluate the practical challenge posed by SpokenUS, the authors measured the Final Slot F1 of voice assistants receiving simulated user interactions across text, cascaded ASR, and native audio modalities. Additionally, interruption detection accuracy was benchmarked on 1,000 streaming turns.

Simulator Input Agent: GPT-4.1-mini (Text) Agent: GPT-4.1-mini (Cascaded) Drop (\(\Delta\)) Agent: Qwen2.5-Omni-7B (Audio)
Qwen2.5-Omni-3B 86.8 83.1 โ†“ 3.7 60.1
SpokenUS (Ours) 73.8 49.5 โ†“ 24.3 56.1
Decision Category Correct Trigger Early Trigger Confused Label Missed Trigger
Turn-end detection 66.0% 10.4% 16.4% 7.2%
Barge-in detection 58.6% 18.0% 11.0% 12.4%

Key Findings

  • High Goal Alignment with Minimal Audio Data: Trained on only ~1,000 hours of speech, SpokenUS achieves a 0.82 GAโ€”on par with the 7B omni baseline (0.80) and vastly outperforming equal-sized models (Qwen2.5-Omni-3B at 0.36; InteractiveOmni-4B at 0.59). Decoupled text-stage grounding effectively imparts TOD policy adherence without needing tens of millions of audio hours.
  • Human-Like Gradual Slot Disclosure: Omni models prematurely saturate slot disclosures within the first 1-2 turns, a deficiency that prompt instructions fail to mitigate. In contrast, SpokenUS exhibits a smooth, human-like linear cumulative slot disclosure curve across the entire conversation.
  • Severe Stress-Testing of Existing Voice Agents: SpokenUS induces a dramatic 24.3-point drop in agent tracking F1 when transitioning from text to cascaded audio (86.8 \(\to\) 49.5), compared to a minor 3.7-point drop under Qwen2.5-Omni-3B. Error analysis confirms voice assistants routinely accept erroneous ASR transcripts of disfluent entities rather than querying confirmation.

Highlights & Insights

  • Proactive Turn-Taking in a Unified Model: By framing streaming turn management as a multi-task prediction head over audio token representations, SpokenUS achieves 240 ms latency decisions, allowing genuine mid-sentence interruptions without external VAD hacks.
  • Transparent Two-Stage Generation: Decomposing speech generation into explicit transcript formulation followed by flow-matching acoustic synthesis guarantees that behavioral tags ([FP], [COR], emotion) are verifiable in text before being rendered prosodically into audio.

Limitations & Future Work

  • English-Only & Clean Acoustic Channel: The current benchmark and models are restricted to English and synthesized under clean acoustic conditions without background environmental noise, reverb, or packet loss.
  • Absence of Simultaneous Overlap: When SpokenUS executes a barge-in, it instantly truncates the incoming assistant audio; true duplex continuous overlapping speech and non-verbal vocalizations (laughter, coughing) remain unmodeled.
  • vs. Text-Based TOD User Simulators (e.g., Agenda-based US, TDM-LLM): Text simulators lack timing and acoustic dimensions entirely; appending an external TTS module still cannot simulate spontaneous interruptions or prosodic feedback loops.
  • vs. General-Purpose Omni Models (e.g., Qwen2.5-Omni, GLM-4-Voice): General omni models lack TOD grounding, frequently drift into helpful assistant personas, and dump goal slots prematurely, whereas SpokenUS enforces role consistency and human-like gradual communication.

Rating

  • Novelty: โญโญโญโญโญ First dedicated full-duplex spoken user simulator for task-oriented dialogue with an end-to-end turn-taking head.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive closed-loop interactive evaluations against five competitive omni baselines, human MOS blind tests, and adversarial agent testing.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, disciplined notation, and detailed documentation of operating modes and training schedules.
  • Value: โญโญโญโญโญ Provides an indispensable and scalable spoken benchmark/evaluation sandbox for testing next-generation conversational voice agents.