Bulbul: A Dataset for Dialectal Arabic Speech Recognition¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Audio and Speech Processing
Keywords: Dialectal Arabic ASR, Speech Recognition Benchmark, Accented Speech, Multi-Dialect Corpus, Zero-Shot Evaluation
TL;DR¶
BULBUL is a community-driven multi-dialect Arabic automatic speech recognition dataset spanning 275 native speakers across 11 Arab countries, providing 71.59 hours of speech covering 16 dialects/sub-dialects alongside accented Modern Standard Arabic and Classical Arabic subsets with two-level human verification.
Background & Motivation¶
Arabic automatic speech recognition (ASR) is confronted by profound linguistic complexity rooted in pervasive diglossia and extensive regional dialectal fragmentation. While Modern Standard Arabic (MSA) serves as the formal literary standard for education, law, and high-level media across the Arab world, daily spoken communication takes place in regional vernaculars that diverge sharply across phonetic, phonological, lexical, and morphological dimensions. Existing public speech corpora are predominantly skewed toward a small set of well-resourced dialects such as Egyptian, Saudi, or broad Levantine, leaving major speech communities in Algeria, Sudan, Yemen, and Tunisia severely underrepresented. Consequently, modern multilingual ASR foundation models exhibit catastrophic performance degradation when deployed in real-world dialectal and community-conversational settings.
This data scarcity is compounded by persistent methodological dilemmas in existing Arabic speech resources. Large-scale corpora crawled from broadcast media, parliamentary proceedings, or social video platforms inevitably suffer from domain biases, acoustic reverberation, overlapping speech, and unreliable automated or crowd-sourced pseudo-transcriptions. Conversely, controlled in-lab reading corpora, while clean, typically rely on scripted formal prompts that erase natural conversational idioms, colloquial syntax, and spontaneous phonetic elisions. Crucially, prior benchmarks fail to isolate the confounding influences of regional accent versus dialectal vocabulary: there has been no dedicated multi-country benchmark enabling direct comparison between dialectal vernacular speech and standardized formal Arabic (MSA and Classical Arabic) articulated by the exact same regional speaker groups.
To resolve the dual bottlenecks of geographic dialect coverage and annotation fidelity, the authors orchestrate a community-driven initiative across 11 Arab countries with a unified recording and verification protocol. By collecting naturalistic colloquial texts alongside parallel formal literary readings and enforcing rigorous multi-stage quality control, the project establishes an authoritative ground truth for Arabic speech processing. Core idea: curate a 71.59-hour multi-dialect speech dataset spanning 11 countries and 16 fine-grained sub-dialects using two-tier human verification and standardized accented MSA/CA subsets to systematically map and benchmark the robustness frontier of contemporary ASR across the Arabic linguistic continuum.
Method¶
Overall Architecture¶
The curation and benchmarking pipeline of BULBUL constitutes a distributed, multi-stage engineering workflow structured across dialect selection, multi-source text collection, interactive recording with self-verification, secondary expert cross-verification, text normalization, and zero-shot acoustic benchmarking. Initiated as a pilot across Saudi Arabia and Egypt to stabilize tooling and protocols, the collection expanded to 11 Arab countries. To address intra-country variance, Saudi Arabia is partitioned into six sub-dialects (Hijazi, Najdi, Southern, Eastern, and Hijazi-Bedouin) and Yemen into two (Sana'ani and Ta'izzi). Textual prompts are assembled from public datasets, social media comments, and private instant messaging, then categorized across an 11-class domain taxonomy via GPT-5-mini. Participants record prompts through a custom Gradio web tool featuring immediate audio self-inspection, followed by country-native reviewer cross-validation to filter acoustic degradation, dialect drift, and transcription discrepancies before standardized zero-shot ASR inference.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-Source Dialectal Text Collection & Domain Classification"] --> B["Multi-Stage Text Preprocessing & Sensitive Content Filtering"]
B --> C["Stratified Distributed Audio Recording & Real-Time Self-Verification"]
C --> D["Native Speaker Double-Blind Cross-Verification & Noise Partitioning"]
D --> E["Standardized Text Normalization & Acoustic Zero-Shot Benchmarking"]
Key Designs¶
1. Multi-Source Dialectal Text Collection & Domain Classification: Grounding natural colloquial idioms across diverse semantic genres
Previous corpora rely heavily on scripted news texts, missing the colloquial lexicon and flexible syntax characteristic of natural spontaneous speech. BULBUL remedies this by sourcing text from three distinct tiers: established regional dialectal corpora (such as Casablanca, SADA, and MADAR), public social media discussions on Facebook, X, and YouTube, and voluntary, explicitly consented private conversational logs from WhatsApp and Telegram. To ensure balanced semantic coverage rather than unconstrained topical drift, an LLM-assisted domain classification schema using GPT-5-mini categorizes sentences across 11 predefined domains (Social, Politics, Economy, Religion, Health, Education, Technology, Entertainment, Sports, Crime, and Other). Prompts require structured confidence ratings, and all classifications scoring below 0.70 undergo mandatory human re-adjudication, ensuring that while natural social communication constitutes the core (65.4%), substantive topical breadth is preserved across every participating nation.
2. Multi-Stage Text Preprocessing & Sensitive Content Filtering: Purging non-standard orthographic noise and dialectal inconsistencies
Raw online texts and informal chat transcripts are filled with typographical inconsistencies, informal slang, emojis, and sensitive discourse. BULBUL implements an automated multi-stage text cleaning tool deployed via Gradio to sanitize plain text inputs. The automated module removes HTML tags, parenthetical side-notes, emojis, ASCII emoticons, non-Arabic script, Arabic diacritics (Tashkeel), elongation characters (Tatweel), and excessive character repetitions, while mapping varied whitespaces and punctuation marks to standard conventions. Following automated cleaning, country-specific team leads execute manual screening to eliminate politically sensitive materials, hate speech, offensive slang, and cross-dialectal intrusions. For the standardized Tarjamat subset used to evaluate accented MSA and Classical Arabic (CA), written numeric digits are systematically removed to prevent subjective verbalization discrepancies during recording.
3. Stratified Distributed Audio Recording & Real-Time Self-Verification: Empowering speaker autonomy and ensuring standardized capture
Unsupervised crowd-sourced audio recording often suffers from severe reverberation, clipping, and environmental noise. BULBUL deploys an interactive web recording suite engineered with Gradio, segmenting prompts by country and sub-dialect. Across participating countries, 275 native speakers (160 male, 115 female) were recruited through localized institutional networks and formal community volunteer platforms (notably the Saudi National Volunteer Platform, incentivized via accredited volunteer service hours). The recording platform presents strict acoustic instructionsโmandating quiet environments, elimination of competing background voices, and preference for external headsetsโalongside instructional videos. The interface embeds an active self-verification loop where speakers review their recorded waveform, trim silences, re-record mispronunciations, or utilize a Skip function for challenging prompts prior to saving.
4. Native Speaker Double-Blind Cross-Verification & Noise Partitioning: Securing verified transcriptions while harvesting in-the-wild noise baselines
Speaker self-monitoring cannot fully prevent dialect drift or subtle phonetic mismatches. BULBUL institutes an external verification mechanism where recordings are independently reviewed by native speakers from the corresponding country. Reviewers utilize a dedicated auditing tool operating on a standardized decision tree: rejections must be classified into predefined primary failure modes (unclear recording, text-audio mismatch, prolonged silence, dialect mismatch, or other). Unclear recordings require explicit sub-categorization into background noise, secondary speaker interference, noticeable acoustic echo, or low recording amplitude. Crucially, rather than discarding acoustically imperfect utterances, rejected samples with audible environmental noise are curated into a dedicated companion Noise Set, providing a valuable realistic resource for training noise-robust dialectal ASR.
5. Standardized Text Normalization & Acoustic Zero-Shot Benchmarking: Eliminating superficial orthographic bias across model families
Dialectal Arabic lacks a codified orthographic authority, meaning identical phonetic sequences can be written with divergent Alef variants, Ta-Marbuta forms, or dialectal spelling habits. To ensure that reported Word Error Rates (WER) and Character Error Rates (CER) reflect true acoustic and linguistic decoding capability rather than trivial formatting discrepancies, a rigorous evaluation text normalization pipeline is executed on both reference and model-predicted transcripts. The protocol applies Unicode NFKC normalization, strips all diacritics and Tatweel, converts Arabic-Indic numerals to ASCII digits, maps all Alef variants (ุฃ, ุฅ, ุข, ูฑ) to bare Alef (ุง), removes punctuation and special symbols, and collapses repeated whitespace into single space delimiters. This uniform protocol enables rigorous, reproducible zero-shot benchmarking across diverse foundation models including Whisper, SeamlessM4T, MMS, and multi-scale OmniASR.
Loss & Training¶
The empirical evaluation focuses on zero-shot out-of-the-box performance without fine-tuning on BULBUL data, measuring generalization under native decoding defaults. The architectures span two core training objectives: - Autoregressive LLM-Decoded Models (OmniASR-LLM 300M, 1B, 3B, 7B & Whisper): Optimized via sequence-to-sequence cross-entropy over target token sequences \(\mathbf{Y} = (y_1, \dots, y_T)\) conditioned on speech representations \(\mathbf{X}\): $\(\mathcal{L}_{\text{LLM}} = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, \mathbf{X})\)$ - Non-Autoregressive CTC Models (OmniASR-CTC 300M, 1B, 3B, 7B & MMS-1B): Optimized using the Connectionist Temporal Classification loss, integrating over all valid frame-level alignment paths \(\pi \in \mathcal{B}^{-1}(\mathbf{Y})\): $\(\mathcal{L}_{\text{CTC}} = -\log \sum_{\pi \in \mathcal{B}^{-1}(\mathbf{Y})} \prod_{t=1}^{T'} P(\pi_t \mid \mathbf{X})\)$ - Performance Evaluation: Normalized Word Error Rate (WER) is computed via Levenshtein distance on tokenized words: $\(\text{WER} = \frac{S + D + I}{N} \times 100\%\)$ where \(S\), \(D\), and \(I\) denote substitution, deletion, and insertion counts relative to the reference length \(N\).
Key Experimental Results¶
Main Results¶
The table below details the zero-shot performance across evaluated multilingual speech recognition systems on the dialectal Arabic test split of BULBUL, grouped by regional dialect clusters (WER/CER in %):
| Model | Northwest African (DZ / TN / MA) | Arabian Peninsula (UAE / KSA / YE) | Levantine (JO / SY / PS) | Nile Valley (EG / SD) | Overall Mean WER / CER |
|---|---|---|---|---|---|
| Whisper-V3 | 80.5/42.1 (DZ) / 48.4/11.7 (TN) | 46.8/13.9 (UAE) / 26.0/9.7 (KSA) | 34.9/8.6 (JO) / 29.6/9.3 (PS) | 61.6/29.8 (EG) / 65.6/38.5 (SD) | 53.8 / 22.9 |
| Whisper-V3-Turbo | 88.2/46.0 (DZ) / 51.2/12.7 (TN) | 60.2/19.9 (UAE) / 33.3/12.0 (KSA) | 39.6/10.7 (JO) / 34.7/13.0 (PS) | 81.6/44.3 (EG) / 83.8/41.7 (SD) | 63.2 / 26.8 |
| SeamlessM4T | 69.5/28.0 (DZ) / 45.2/10.6 (TN) | 45.9/12.7 (UAE) / 27.4/10.3 (KSA) | 34.4/8.1 (JO) / 26.1/7.2 (PS) | 43.3/16.3 (EG) / 70.9/30.1 (SD) | 49.8 / 16.6 |
| MMS-1B | 88.5/31.7 (DZ) / 70.7/18.8 (TN) | 72.9/21.0 (UAE) / 66.8/22.3 (KSA) | 71.9/20.8 (JO) / 62.4/18.8 (PS) | 84.9/34.0 (EG) / 79.1/27.5 (SD) | 77.1 / 25.1 |
| OmniCTC-1B | 58.5/15.9 (DZ) / 46.2/12.2 (TN) | 47.2/13.7 (UAE) / 26.9/7.9 (KSA) | 39.0/9.7 (JO) / 26.0/7.6 (PS) | 52.6/20.3 (EG) / 65.8/28.7 (SD) | 49.5 / 16.6 |
| OmniLLM-1B | 60.0/19.2 (DZ) / 43.8/10.5 (TN) | 46.5/13.3 (UAE) / 22.3/7.0 (KSA) | 33.3/7.7 (JO) / 19.8/5.4 (PS) | 45.1/17.0 (EG) / 61.0/22.5 (SD) | 45.1 / 13.9 |
| OmniLLM-3B | 58.1/19.1 (DZ) / 42.7/8.9 (TN) | 43.6/11.8 (UAE) / 19.7/6.5 (KSA) | 31.6/6.8 (JO) / 20.0/5.5 (PS) | 43.1/16.8 (EG) / 59.3/20.5 (SD) | 43.6 / 13.0 |
| OmniLLM-7B (Best) | 56.1/18.4 (DZ) / 42.7/10.0 (TN) | 43.0/10.9 (UAE) / 19.6/6.5 (KSA) | 30.8/6.8 (JO) / 17.8/4.8 (PS) | 40.2/14.3 (EG) / 56.5/19.1 (SD) | 41.7 / 12.7 |
Ablation Study¶
To decouple the acoustic impact of regional accent from lexical and morphological dialectal shifts, models were evaluated on the Accented MSA and Classical Arabic (CA) test split recorded by speakers from the same dialectal origins. The table below highlights the performance and the dialect-to-accent robustness gap (\(\Delta\)WER):
| Evaluated System | Accented MSA/CA Mean WER / CER (%) | Overall Dialect-to-Accent Gap (\(\Delta\)WER %) | Representative Regional Disparity (DZ / YE Gap) | Architectural Robustness Behavior |
|---|---|---|---|---|
| Whisper-V3 | 19.2 / 8.0 | +34.6 | DZ: +65.1 / YE: +53.6 | Strong on standard speech, but struggles heavily with colloquial Maghrebi idioms |
| Whisper-V3-Turbo | 20.0 / 7.5 | +43.2 | DZ: +71.8 / YE: +64.0 | Pruning and distillation exacerbate error propagation on non-standard vocabulary |
| SeamlessM4T | 14.4 / 5.4 | +35.4 | DZ: +58.3 / YE: +52.2 | High acoustic competence on MSA; notable degradation on North African vernaculars |
| MMS-1B | 33.8 / 9.7 | +43.3 | DZ: +57.9 / YE: +56.7 | Lacks autoregressive language model guidance, leading to severe phonemic confusion |
| OmniCTC-1B | 18.3 / 6.0 | +31.2 | DZ: +45.1 / YE: +49.5 | CTC decoding performs reliably at 1B scale but degrades on larger parameter models |
| OmniLLM-1B | 14.2 / 5.3 | +30.9 | DZ: +49.9 / YE: +45.2 | Autoregressive language modeling provides robust contextual error recovery |
| OmniLLM-3B | 13.7 / 5.1 | +29.9 | DZ: +47.9 / YE: +46.7 | Progressive reduction in context drift and dialectal word substitutions |
| OmniLLM-7B | 13.3 / 4.8 | +28.4 | DZ: +47.2 / YE: +45.1 | Lowest overall error rate and smallest robustness gap across regional accents |
Key Findings¶
- Contextual decoders powered by LLMs demonstrate superior dialectal resilience: The OmniLLM model family consistently outstrips pure CTC baselines and Whisper variants. OmniLLM-7B achieves the best overall performance with 41.7% dialectal WER and 13.3% accented MSA/CA WER, demonstrating that deep language model priors can recover phonetic ambiguity caused by regional pronunciation shifts.
- Dialectal divergence drives the performance gap rather than accent alone: Across all evaluated systems, error rates drop dramatically when speakers read standardized MSA/CA text compared to colloquial dialectal text. For Algerian and Yemeni Arabic, the gap between dialectal and accented speech reaches 45% to 70% WER, proving that lexical substitution, phonotactic restructuring, and colloquial morphology are far greater hurdles for modern ASR than regional accent variation alone.
- Scaling challenges in CTC architectures: Within the OmniCTC family, moving from 300M (65.7% WER) to 1B (49.5% WER) yields substantial gains, but scaling further to 3B (53.3% WER) and 7B (51.8% WER) causes performance to plateau or slightly degrade. This highlights optimization bottlenecks when applying non-autoregressive CTC decoding to massive multi-dialect speech corpora without language model rescoring.
Highlights & Insights¶
- Strictly controlled dialect vs. accent experimental decoupling: By capturing both localized colloquial speech and standardized MSA/CA readings from the exact same cohort of 275 native speakers across 11 nations, the benchmark provides the first clean empirical baseline to separate pure acoustic accent variance from deep lexical/structural dialectal divergence.
- Recycling rejected recordings into a realistic noise benchmark: By preserving and categorizing audio rejected during secondary human verification due to acoustic interference (urban noise, reverberation, overlapping speakers), the creators establish a high-value real-world noise evaluation set rather than discarding imperfect data.
- Fine-grained sub-dialectal geographic resolution: Moving beyond coarse country-level labels, the dataset isolates six sub-dialects across Saudi Arabia and two in Yemen, illustrating how local phonological processes (such as Najdi vs. Southern Saudi consonant metathesis in words like /gahwa/ vs. /ghawa/) impact acoustic modeling.
Limitations & Future Work¶
- Demographic skew due to voluntary community recruitment: Specific subsets display demographic imbalances reflecting volunteer availability; Sudan comprises exclusively male participants while the UAE subset is ~89.5% female, limiting nuanced gender-fairness modeling across all territories.
- Neutral sentence-level reading style: Due to prompt-based recording without conversational narrative context, participants tended toward neutral, formal reading deliveries, limiting applicability to high-emotion or unconstrained conversational ASR tasks.
- Future directions: Expanding coverage to rural and hard-to-access sub-dialects in North Africa and the Levant, integrating multi-turn spontaneous dialogue, and developing dialect-conditioned adapter tuning for large speech foundation models.
Related Work & Insights¶
- vs QASR / MGB-2 / MASC: Prior large-scale multi-dialect resources draw predominantly from Al Jazeera broadcasts or YouTube videos, introducing severe professional media domain bias, room acoustics, and uneven transcription fidelity. BULBUL is entirely community-recorded with rigorous double-blind human verification and explicit sub-dialect demarcation.
- vs Casablanca: Casablanca made strides in multi-dialect Arabic ASR but lacked an explicit parallel accented MSA/CA subset spoken by the identical demographic cohort. BULBUL enables rigorous ablation of accent vs. dialectal vocabulary effects across 12 modern foundation model architectures.
Rating¶
- Novelty: โญโญโญโญโ Pioneering community-driven multi-dialect Arabic ASR benchmark with fine-grained sub-dialect mapping and parallel accented formal speech.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive zero-shot evaluation across 12 foundation models spanning Whisper, Seamless, MMS, and multi-scale CTC/LLM decoders.
- Writing Quality: โญโญโญโญโญ Rigorously documented dataset construction protocols, lexical diversity statistics, and phonological failure analyses.
- Value: โญโญโญโญโญ A critical public asset addressing linguistic disparity and advancing inclusive spoken language technologies for the Arabic-speaking world.