CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognition¶

Conference: NeurIPS 2026
arXiv: 2502.18913
Paper: https://arxiv.org/abs/2502.18913
Area: Audio & Speech
Keywords: Code-Switching, Automatic Speech Recognition, Spontaneous Dialogue Dataset, Multilingual Speech Modeling, Benchmark Evaluation

TL;DR¶

To address the scarcity of natural, full-length conversational speech resources in code-switching, this paper presents CS-Dialogue, a 104-hour spontaneous Mandarin-English dialogue dataset featuring 200 speakers with comprehensive transcriptions and multi-paradigm ASR benchmarking.

Background & Motivation¶

Code-switching (CS)—the alternation between two or more languages within a single conversation or utterance—is a ubiquitous linguistic phenomenon in multilingual communities. For automatic speech recognition (ASR) systems, code-switching introduces severe acoustic and language modeling challenges: models must negotiate abrupt transitions across distinct phonetic inventories in a shared acoustic space, while simultaneously capturing intra-utterance syntactic and lexical switching patterns without explicit language boundary indicators. Traditional ASR models trained predominantly on monolingual corpora suffer dramatic degradation in code-switched environments due to acoustic feature mismatch and severe language model confusion.

Despite the urgent demand for robust Mandarin-English bilingual speech systems, existing open-source speech resources remain inadequate for training and evaluating models under genuine conversational dynamics. Most available corpora suffer from critical bottlenecks: first, corpora such as CECOS, OC16-CE80, and DOTA-ME-CS predominantly consist of read speech from pre-scripted sentences, lacking natural conversational disfluencies, pauses, overlaps, and spontaneous cadence; second, large-scale open-source datasets like TALCS (587 hours) are restricted to online tutoring scenarios with rigid grammatical and discourse structures; third, the few spontaneous conversational corpora are either proprietary and costly (such as SEAME), extremely small (such as ASCEND with only 10.6 hours), or limited to isolated sentence snippets rather than full-length dialogue sessions, destroying multi-turn acoustic and conversational context.

This severe data deficit impedes comprehensive investigation into how foundation models perform in natural bilingual multi-turn interactions. To overcome these limitations, researchers from Nankai University and Beijing Academy of Artificial Intelligence constructed CS-Dialogue. Core idea: build the largest publicly available 104-hour spontaneous Mandarin-English code-switching dataset containing 200 speakers across 100 full-length dialogues, featuring progressive topic-guided recording protocols, high-fidelity acoustic-preserving transcription, and multi-paradigm ASR benchmarks across scratch-trained and foundation models.

Method¶

Overall Architecture¶

The development of CS-Dialogue is structured around three foundational principles: spontaneous interaction fidelity, full-length continuous dialogue recording, and high-precision non-lexical transcription. The data engineering pipeline encompasses speaker recruitment and screening, topic-guided paired dialogue sessions, rigorous verbatim transcription with acoustic event annotation, speaker-independent data partitioning, and multi-paradigm ASR benchmark evaluations.

The following flowchart outlines the entire dataset creation and benchmarking process:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Paired Bilingual Speakers<br/>200 proficient Mandarin-English speakers"] --> B["Progressive Multi-Turn Dialogue Acquisition<br/>Mandarin 8min → Mixed 6min → English 6min"]
    B --> C["High-Fidelity Verbatim & Non-Lexical Annotation<br/>Exact transcriptions + 5 acoustic tags + QC review"]
    C --> D["Speaker-Independent Partitioning<br/>Train: 140 spk / Dev: 30 spk / Test: 30 spk"]
    D --> E["Multi-Paradigm ASR Benchmarking<br/>From-scratch / Zero-shot / Fine-tuning"]

Key Designs¶

1. Progressive Multi-Turn Dialogue Acquisition: Balancing Natural Interaction with Code-Switching Density In unconstrained spontaneous conversation recording, bilingual speakers naturally tend to remain in their dominant native tongue, yielding sparse code-switching occurrences; conversely, imposing rigid switching directives disrupts spontaneous conversational flow and authenticity. To achieve dense language switching while preserving conversational realism, the authors established a structured paired dialogue framework. Each recording session paired two native Chinese speakers with demonstrated English proficiency (IELTS $\ge$ 6.0 or TEM-4, audited through pre-recording auditions) in a quiet acoustic setting using smartphone microphones (16 kHz, 16-bit mono PCM WAV). Discussions spanned seven common daily topics (Personal, Entertainment, Technology, Education, Job, Philosophy, Sports), structured into ~20-minute sessions progressing through: 8 minutes of Mandarin, 6 minutes of code-switching, and 6 minutes of English. A timekeeper and observer tracked pacing and adherence without interfering in natural turn-taking or disfluencies, resulting in rich, natural intra-sentential and clause-boundary code-switching patterns (such as "YOU KNOW 就" and "PLAY 麻将").

2. High-Fidelity Verbatim & Non-Lexical Annotation: Preserving Spoken Phenomena and Acoustic Artifacts Real spontaneous speech is permeated with disfluencies, false starts, word repetitions, and non-verbal physiological sounds; overly aggressive cleaning or text normalization cripples ASR robustness when models encounter real conversational inputs. CS-Dialogue formulated a strict transcription protocol: first, strict word-count fidelity was enforced, prohibiting transcribers from adding or omitting words for perceived grammatical polish; second, stuttering and sound repetitions (e.g., "放放假") were preserved verbatim, and truncated syllables were transcribed using closest homophones (e.g., "fu-fang4-jia4" transcribed as "夫放假"), while Arabic numerals were transcribed as contextual Chinese characters ("七幺幺") or English words ("Seven Eleven"); third, regional Chinese accent variations (e.g., retroflex vs. non-retroflex, /f/ vs. /h/) were faithfully preserved without standardization; fourth, five standardized acoustic event markers were embedded: unintelligible speech (**), hesitation fillers (<FIL/>), speaker vocal noises (<SPK/>, including laughter and throat clearing), environmental noise (<NON/>), and non-target speaker speech (<NPS/>). An independent quality control team reviewed all transcriptions iteratively to guarantee high annotation consistency.

3. Speaker-Independent Partitioning & Multi-Paradigm Benchmarking: Establishing a Rigorous CS-ASR Baseline To prevent data contamination and ensure valid assessment of out-of-speaker generalization, CS-Dialogue strictly implements speaker-independent partitioning across all three splits: the training set contains 140 speakers (68.97 hours, 26,428 utterances), the development set contains 30 speakers (18.30 hours, 6,196 utterances), and the test set contains 30 speakers (16.74 hours, 6,293 utterances). The standard Mixed Error Rate (MER) is adopted as the primary metric, computing edit distance over Chinese characters and English words: $$MER = \frac{S + D + I}{N_{\text{total}}} \times 100\%$$ where $S, D, I$ represent substitutions, deletions, and insertions, and $N_{\text{total}}$ is the reference length. The benchmarking suite covers three distinct modeling regimes: WeNet-based from-scratch training (Transformer, Branchformer, Conformer), zero-shot evaluation on leading speech foundation models (Whisper Large-V2, Qwen2-Audio, Paraformer, SenseVoice-Small), and full supervised fine-tuning across multiple scales of Whisper (Tiny, Base, Small, Medium), mapping out current speech recognition bottlenecks on spontaneous code-switching.

A Worked Example¶

Consider a representative segment from the Education topic: 1. Audio Capture: Two participants debate master's degree graduation requirements in a 10.6-second continuous 16 kHz audio segment. 2. Annotation & Alignment: Speakers produce fluent intra-sentential switching, transcribed verbatim with acoustic fidelity: "那是你们这个 group 自己去想一个 topic 呢，还是这个 professor 会提供一些他的 project 来支撑你们的毕业论文" 3. Comparative ASR Decoding: - Zero-shot general foundation models (such as un-tuned Whisper) frequently suffer language-bias drift on this segment, hallucinating English translations of the Mandarin frame and resulting in high error rates. - Fine-tuned models or Conformer with Attention Rescoring accurately identify the phonetic transitions between English nouns and Mandarin function words, outputting exact aligned tokens.

Loss & Training¶

For models trained from scratch (Transformer, Branchformer, Conformer), training utilizes a hybrid loss combining CTC (Connectionist Temporal Classification) and AED (Attention-based Encoder-Decoder): $$\mathcal{L}_{\text{total}} = \alpha \mathcal{L}_{\text{CTC}} + (1 - \alpha) \mathcal{L}_{\text{AED}}$$ Models are trained with dynamic batching, an initial learning rate of $1.00 \times 10^{-3}$ for 150 epochs across four NVIDIA GTX 3090 GPUs. Decoding strategies evaluated include Greedy decoding, CTC Beam Search, Attention Decoding, and Attention Rescoring.

For fine-tuning Whisper models (Tiny, Base, Small, Medium), training runs for 20 epochs with dynamic batches capped at 60,000 frames per batch and an initial learning rate of $1.00 \times 10^{-5}$.

Key Experimental Results¶

Main Results¶

The table below reports the test set performance of models trained from scratch under various decoding strategies.

Model	# Params	Decoding Strategy	CER (%)	WER (%)	MER (%)
Transformer	29M	Greedy	22.56	45.34	27.21
Transformer	29M	Beam Search	22.24	45.19	27.01
Transformer	29M	Attention Rescoring	21.60	43.44	26.06
Branchformer	29M	Greedy	18.86	39.16	23.01
Branchformer	29M	Beam Search	18.78	39.20	22.95
Branchformer	29M	Attention Rescoring	18.29	37.55	22.23
Conformer	31M	Greedy	15.91	33.67	19.54
Conformer	31M	Beam Search	15.88	33.60	19.50
Conformer	31M	Attention Rescoring	15.45	32.36	18.91

Ablation & Analysis Studies¶

Evaluation of pre-trained foundation models in zero-shot mode on the CS-Dialogue test set reveals error decompositions across substitution (S), deletion (D), and insertion (I):

Model	# Params	CER (%)	WER (%)	S (%)	D (%)	I (%)	Overall MER (%)
Whisper Large-V2	1,550M	10.70	31.11	6.00	7.60	1.69	15.29
Qwen2-Audio	8.2B	7.15	19.82	4.32	1.82	3.62	9.76
Paraformer	220M	3.70	32.02	6.30	0.98	2.37	9.65
SenseVoice-Small	234M	4.42	15.57	3.44	1.42	1.85	6.71

Furthermore, the supervised fine-tuning impact across different Whisper scales is summarized below:

Whisper Scale	# Params	Zero-shot CER (%)	Zero-shot WER (%)	Zero-shot MER (%)	Fine-tuned CER (%)	Fine-tuned WER (%)	Fine-tuned MER (%)	MER Reduction
Whisper Tiny	38M	27.83	41.69	31.11	19.24	29.64	21.38	-9.73%
Whisper Base	74M	19.90	37.21	23.90	15.36	27.20	17.80	-6.10%
Whisper Small	244M	12.82	30.81	16.76	7.51	16.09	9.26	-7.50%
Whisper Medium	769M	11.34	32.57	15.88	6.12	13.02	7.53	-8.35%

Key Findings¶

Substitution Errors Heavily Dominate: Across both from-scratch and foundation models, substitution (S) constitutes the largest fraction of MER, underscoring that models frequently confuse cross-lingual phoneme boundaries or generate phonetically similar substitutions.
Model Scale Alone Does Not Guarantee CS Robustness: Despite 1.55B parameters and 680,000 hours of weakly supervised multilingual pre-training, Whisper Large-V2 achieves a zero-shot MER of 15.29% (WER 31.11%), markedly worse than the 234M parameter SenseVoice-Small (6.71%). Qualitative analysis reveals that Whisper occasionally translates input Chinese speech into English instead of transcribing verbatim, causing catastrophic deletion and substitution penalties.
Discourse Complexity Impacts Topic Error Rates: Error rates vary widely across topics. "Philosophy" and "Sports" exhibit the highest MERs due to uncommon technical terminology and extended sentence lengths (Philosophy test utterances average 12.95 seconds), whereas "Job" and "Technology" yield significantly lower error rates across all architectures.
In-Domain Fine-Tuning Produces Massive Gains: Fine-tuning Whisper Medium on CS-Dialogue for only 20 epochs slashes its MER from 15.88% to 7.53% (a >52% relative improvement), proving the critical necessity of domain-specific spontaneous code-switching corpora.

Highlights & Insights¶

Fills the Full-Dialogue Conversational Void: CS-Dialogue provides the community with 104 hours of unscripted multi-turn conversational speech with full transcriptions, enabling research on context-dependent conversational ASR.
Identifies Multi-Task Objective Drift: The study highlights an unintended failure mode in multitask foundation models like Whisper, where code-switching input triggers spontaneous speech-to-text translation (S2TT) behavior instead of ASR transcription.
Demonstrates Superiority of Compact Non-Autoregressive Models: SenseVoice-Small and Paraformer achieve state-of-the-art CER and MER with compact footprints, offering high utility for real-time edge and server deployments.

Limitations & Future Work¶

Restricted Language Pair: The dataset is solely focused on Mandarin-English code-switching and does not include Chinese regional dialects (e.g., Cantonese, Sichuanese) or other language pairings.
Bilingual Background Asymmetry: All 200 participants are native Chinese speakers with high English proficiency; there are no native English speakers code-switching into Chinese.
Acoustic Environment Uniformity: Recordings were conducted in quiet indoor environments using near-field smartphones, lacking real-world acoustic challenges such as background noise, far-field reverberation, and overlapping conversational interruptions.

vs SEAME: SEAME comprises ~30 hours of spontaneous Southeast Asian Mandarin-English conversations, but requires paid licensing and features strong regional Singaporean/Malaysian accents; CS-Dialogue is fully open-source and covers 104 hours across Chinese mainland speakers.
vs TALCS: TALCS offers 587 hours but is strictly collected from online tutoring environments; CS-Dialogue focuses on 7 open everyday conversational themes with richer spontaneous discourse dynamics.
vs ASCEND: While ASCEND provides open spontaneous conversational data, it totals only 10.62 hours with 23 speakers; CS-Dialogue is roughly 10 times larger and establishes a speaker-independent benchmark.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [First open-source 100+ hour full-length spontaneous Mandarin-English conversational CS dataset]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive benchmarking across from-scratch models, 4 foundation models, and fine-tuning sweeps]
Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary dataset documentation, transparent transcription guidelines, and insightful error analysis]
Value: ⭐⭐⭐⭐⭐ [Crucial benchmark resource filling a long-standing gap in code-switching speech recognition research]