Skip to content

Unsupervised Speech Recognition at the Syllable Level

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/cactuswiththoughts/SylCipher
Area: Audio & Speech
Keywords: Unsupervised Speech Recognition, Syllable-Level Modeling, Masked Language Model, Distribution Matching, Cross-Modal Decipherment

TL;DR

SylCipher introduces the first syllable-level unsupervised speech recognition system that jointly segments continuous speech into discrete syllable tokens and aligns them with unpaired text via information-constrained masked language modeling and positional marginal matching, completely bypassing G2P phoneme dictionaries and GAN instability while outperforming prior G2P-free systems by up to 40% relative CER.

Background & Motivation

Unsupervised automatic speech recognition (UASR) seeks to learn an acoustic-to-text transcriber using entirely unpaired speech and text corpora. It represents a vital frontier for expanding speech technology to the thousands of low-resource languages in the long-tail distribution and serves as a fundamental benchmark for unpaired cross-modal representation learning. However, virtually all competitive prior UASR systems are formulated at the phoneme level. This paradigm relies heavily on grapheme-to-phoneme (G2P) converters or comprehensive pronunciation dictionaries to map text into phonetic sequences. In many low-resource and endangered languages, orthographies may omit crucial vowel markings, exhibit regional dialectal drift, or lack standardized phonetic converters entirely; constructing and maintaining such G2P lexicons requires prohibitive linguistic labor. When G2P is unavailable, phoneme-based UASR degrades catastrophically due to acute speech-text token mismatches.

Switching to word-level modeling avoids G2P dependencies, but immediately encounters the unbounded vocabulary of rare words, severe out-of-vocabulary (OOV) rates, and context-dependent word boundary drift that destabilizes distribution matching. Furthermore, prevailing phoneme-level frameworks rely on generative adversarial networks (GANs), which suffer from notorious optimization fragility and mode collapseโ€”failing to converge altogether on languages with strong co-articulation such as Mandarin Chinese. Consequently, existing UASR is trapped between brittle G2P dependencies at the phone level and catastrophic vocabulary explosion at the word level.

This paper tackles this fundamental tension by operating at the syllable level. Syllables possess a finite, bounded inventory that avoids the long-tail token explosion of words, while naturally aligning with written characters in languages like Mandarin and providing sharper acoustic-prosodic boundaries. Core idea: formulate unsupervised speech recognition over discrete syllable units, training a non-adversarial encoder-only masked language model (SylCipher) with differentiable soft pooling, capacity-constrained joint representation learning, and progressive marginal distribution matching to achieve robust cross-modal decipherment.

Method

Overall Architecture

SylCipher is an encoder-only architecture comprising a speech syllabifier, unimodal linear pre-nets, a shared Transformer encoder, and modality-specific linear-softmax post-nets. Given unpaired acoustic feature sequences and text token sequences, the speech syllabifier aggregates continuous frame-level representations into discrete syllable units via differentiable soft-pooling and vector quantization. Both modalities are projected into a common latent space through linear pre-nets and processed by the shared encoder. Training progresses through three stages: weighted masked language modeling (wMLM) with frozen boundaries, joint end-to-end (JE2E) boundary refinement via backpropagated text-matching signals, and positional unigram/skipgram matching (PUSM) for explicit sequence distribution matching. During inference, speech features pass through the syllabifier and pre-net, directly querying the text post-net to decode the predicted syllable sequence without adversarial discriminators.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Continuous Speech Features<br/>HuBERT / XEUS frame-level representations"] --> B["Differentiable Soft-Pooling<br/>Sylber boundary priors + clamped soft assignment"]
    B --> C["Discrete Syllable Quantization<br/>K-means codebook discretization into acoustic tokens"]
    C --> D["Shared Transformer Encoder<br/>Mix-up bottleneck + bidirectional wMLM reconstruction"]
    D --> E["End-to-End Refinement & Marginal Matching<br/>JE2E boundary updates + PUSM N-gram distribution alignment"]
    E --> F["Text Syllable Sequence<br/>Decoded via text post-net (greedy / beam search)"]

Key Designs

1. Differentiable Syllable Soft-Pooling: Stable Acoustic-to-Segment Aggregation

To bridge the temporal resolution gap between dense frame-level features and segmental syllable tokens without the numerical instability of softmax or tanh pooling, SylCipher introduces a localized, clamp-based soft assignment mechanism. An initial unsupervised syllable detector (Sylber) provides boundary priors, and a trainable 5-layer 1D-CNN (TDNN) predicts frame-level boundary probabilities \(b(X) \in [0, 1]^T\). A soft segment assignment mask \(a_{it}(X)\) maps frame \(t\) to segment \(i\):

\[a_{it}(X) := \sigma_\epsilon \left( i - \sum_{\tau \le t} b_\tau(X) \right), \quad \sigma_\epsilon(x) := \epsilon - |\text{clamp}(x, -\epsilon, \epsilon)|\]

Normalizing these weights yields segment-level embeddings \(m_i(X) = \sum_{t=1}^T \frac{a_{it}(X)}{\sum_{\tau=1}^T a_{i\tau}(X)} X_t\). By clamping differences to \([-\epsilon, \epsilon]\) with \(\epsilon=0.5\), the pooling weights achieve exact compact support, ensuring that distant acoustic frames receive zero gradient while preventing cumulative numerical explosion. These continuous segment vectors are then mapped to discrete syllable tokens via a straight-through differentiable K-means vector quantizer.

2. Information-Constrained Masked Language Modeling: Preventing Disjoint Latent Drift

Without paired alignments, training a shared encoder across two modalities easily leads to degenerate representations where speech and text map to disconnected regions of the latent space. SylCipher enforces an information bottleneck by combining a shallow 2-layer Transformer encoder with Gumbel-Softmax vector quantization and random mix-up (replacing encoder hidden states with shared codebook vectors with 0.3 probability).

The model is optimized using weighted masked language modeling (wMLM) across both modalities:

\[\mathcal{L}_{\text{wMLM}} = -\mathbb{E}_{Z \sim \frac{1}{2}p_{\tilde{X}} + \frac{1}{2}p_Y, M \sim p_M} \left[ \ln q_Z(Z_M | Z_{M^c}) + \lambda \ln q_Z(Z_M | Z_M) \right]\]

Masked tokens are sampled via Poisson span masking with an average length of 3.5, balancing masked prediction and full sequence reconstruction (\(\lambda=0.5\)). The paper proves theoretically that under discrete entropy constraints, minimizing this weighted MLM objective minimizes the Kullback-Leibler divergence between true and model marginals \(\mathcal{D}_{\text{KL}}(p_{\tilde{X}} || q_{\tilde{X}}) + \mathcal{D}_{\text{KL}}(p_Y || q_Y)\), guaranteeing exact recovery of the true speech-to-text mapping \(y^*(x)\) up to permutation under mild regularity conditions.

3. Progressive Refinement and Distribution Matching: A Multi-Stage Training Pipeline

To rectify errors in initial unsupervised boundaries without collapsing training, SylCipher adopts an iterative multi-stage schedule. After wMLM pretraining with static boundaries converges, Stage 2 activates joint end-to-end (JE2E) training: gradients from the text distribution matching objective flow back through the differentiable soft-pooler to update the CNN boundary detector, constrained by a soft syllable-count penalty:

\[\mathcal{L}_{\text{JE2E}} = \mathbb{E}_X \left| \sum_{t=1}^T b_t(X) - \hat{b}_t(X) \right|\]

Once boundary refinement and MLM saturate, Stage 3 introduces positional unigram and skipgram matching (PUSM), treating the full dataset as a large batch to directly match low-order N-gram statistics between speech decodings and text distributions, solidifying global word order and sequence alignment.

Loss & Training

The overall composite objective is:

\[\mathcal{L} = \lambda_1 \mathcal{L}_{\text{wMLM}} + \lambda_2 \mathcal{L}_{\text{JE2E}} + \lambda_3 \mathcal{L}_{\text{PUSM}}\]

Optimization is scheduled across three decoupled phases: Stage 1 trains on 8 32-GB V100 GPUs for up to 7000 epochs (Adam, lr \(2 \times 10^{-4}\), update frequency 16, codebook mix-up 0.3); Stage 2 loads the static checkpoint and trains for 700 epochs with update frequency 890 to adjust boundaries; Stage 3 runs PUSM distribution matching with lr \(10^{-3}\) and update frequency 7200 for 2000 steps. A wav2vec 2.0 student is subsequently fine-tuned on pseudo-labels via CTC self-training.

Key Experimental Results

Main Results

Character Error Rate (CER %) on LibriSpeech clean subsets (matched and unmatched text setups) and SpokenCOCO:

Model Token Level G2P Required LibriSpeech Matched CER (โ†“) LibriSpeech Unmatched CER (โ†“) SpokenCOCO CER (โ†“)
wav2vec-U (Baevski et al., 2021) Phone Yes - 13.3* -
wav2vec-U 2.0 (Liu et al., 2023) Phone Yes - 12.2* -
REBORN (Tseng et al., 2024) Phone Yes - 8.3* -
wav2vec-U (Baevski et al., 2021) Char. No 35.6 43.3 45.0 / 45.2
REBORN (Tseng et al., 2024) Char. No 37.8 76.6 -
JSTTI (Ni et al., 2025) Word No 81.4 81.1 64.5 / 64.5
PUSM (Wang et al., 2023c) Syllable No 35.5 54.2 41.5 / 41.3
SylCipher (Ours, Sylber+JE2E+PUSM) Syllable No 21.8 35.9 23.4 / 26.8
SylCipher + Student Self-Training Syllable No 17.5 33.3 13.9 / 17.6

Cross-lingual and low-resource recognition results across Mandarin AISHELL-3, Multilingual LibriSpeech (MLS), and low-resource languages (PER / CER %):

Language & Dataset Metric Type wav2vec-U (Baseline) PUSM (Baseline) SylCipher (Ours) SylCipher + Self-Training
AISHELL-3 Mandarin (w/o tone) PER (โ†“) 74.9 (Phone) 28.4 (Syllable) 26.9 (Syllable) 15.3 (Syllable)
AISHELL-3 Mandarin (with tone) PER (โ†“) 76.2 (Phone) 26.5 (Syllable) 24.9 (Syllable) 12.2 (Syllable)
MLS German (de) CER (โ†“) 32.5 (Phone G2P) / 42.9 (Char) 35.1 (Syllable) 32.3 (Syllable) -
MLS Dutch (nl) CER (โ†“) 40.2 (Phone G2P) / 50.6 (Char) 38.2 (Syllable) 35.6 (Syllable) -
MLS French (fr) CER (โ†“) 39.8 (Phone G2P) / 72.6 (Char) 43.6 (Syllable) 41.7 (Syllable) -
Cantonese MDCC PER (โ†“) 75.7 (Phone) 36.6 (Syllable) 35.0 (Syllable) 23.5 (Syllable)
Taigi SuiSiann PER (โ†“) 69.0 (Phone) 37.3 (Syllable) 35.5 (Syllable) 26.5 (Syllable)

Ablation Study

Ablation on training stages, pooler designs, and adversarial alternatives on LibriSpeech:

Configuration / Ablation LibriSpeech Matched CER (โ†“) LibriSpeech Unmatched CER (โ†“) Note
SylCipher (Forced Alignment oracle) 38.5 46.4 Ideal upper-bound syllable boundaries
SylCipher (Sylber static boundaries) 43.5 48.6 Stage 1 wMLM only
SylCipher (Sylber + JE2E) 39.2 46.8 Stage 2 boundary refinement (-4.3% CER)
SylCipher (Sylber + JE2E + PUSM) 21.8 35.9 Stage 3 marginal matching (-17.4% CER)
Pooler comparison: Clamp (Ours) Benchmark (SER 25.5) - Exact compact support prevents boundary drift
Pooler comparison: Softmax Substantial drop (+6% to 36% SER) - Weight smearing blurs boundary precision
Pooler comparison: Tanh Substantial drop - Gradient explosion and numerical divergence
Adversarial baseline: Syllabic GAN 115.7 - 126.5 SER - Severe mode collapse across all hyper-parameters

Key Findings

  • Syllables represent the optimal G2P-free unit: Character-based baselines degrade severely without G2P (wav2vec-U reaches 43.3% CER; REBORN collapses to 76.6% unmatched), while word-level JSTTI fails to converge past 81% CER. SylCipher achieves 21.8% (17.5% with student), reducing relative CER by 40% over G2P-free baselines.
  • Stage complementarity: JE2E boundary fine-tuning improves boundary detection F1 from 83.4% to 86.1% on LibriSpeech, yielding a 4.3% CER gain; PUSM provides the largest boost by explicitly aligning global N-gram marginal distributions (a 44% relative gain over JE2E in the matched setup).
  • Overcoming adversarial collapse: On Mandarin AISHELL-3, GAN-based models fail completely due to co-articulation (PER > 74%), while SylCipher achieves 24.9% (12.2% with self-training). Direct comparison shows Syllabic GANs produce >115% SER, whereas SylCipher maintains 25.5% SER, proving non-adversarial masked modeling is vastly more stable.

Highlights & Insights

  • First syllable-level UASR paradigm: Exploits the bounded nature of syllable inventories and their natural correspondence to spoken units, unlocking G2P-free unsupervised speech recognition for unwritten or low-resource languages.
  • Clamped soft-pooling operator: The \(\sigma_\epsilon\) formulation creates a localized, compact-support weighting that stabilizes gradient flow back to the boundary detector without the tail smearing of softmax.
  • Theoretical identifiability: Proves that under discrete entropy constraints, bi-directional masked language modeling uniquely recovers the true speech-to-text mapping up to permutation, providing formal justification for unsupervised decipherment.

Limitations & Future Work

  • Orthographic reliance on vowels: Languages using abjad scripts (e.g., Arabic and Hebrew) omit vowels in standard text, which creates challenges for rule-based text syllabifiers like Pyphen+.
  • Multi-stage training complexity: Training requires sequential transitions across static wMLM, JE2E boundary tuning, and large-batch PUSM matching; consolidating this into a single end-to-end loss remains open.
  • Domain mismatch sensitivity: Performance degrades significantly when text comes from out-of-domain sources (e.g., CER increases to 75.6% when mixing 50% image caption text), highlighting the need for greater domain robustness.
  • vs wav2vec-U / wav2vec-U 2.0: Relies on phoneme units, expert G2P dictionaries, and adversarial training that frequently fails to converge on non-alphabetic languages; SylCipher completely removes G2P reliance and uses stable non-adversarial masked modeling.
  • vs JSTTI (Ni et al., 2025): Evaluated non-adversarial UASR at the word level, but encountered insurmountable OOV rates and vocabulary sparsity (CER > 80%); SylCipher demonstrates that syllable units resolve the vocabulary explosion while preserving segmental tractability.

Rating

  • Novelty: โญโญโญโญโญ First formulation of syllable-level UASR with rigorous identifiability proofs and non-adversarial optimization.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluated across six corpora spanning English, Mandarin, Romance, Germanic, and regional dialects under both matched and unmatched settings.
  • Writing Quality: โญโญโญโญโญ Exceptional clarity, combining sound linguistic motivation, clean mathematical formulation, and thorough empirical validation.
  • Value: โญโญโญโญโญ Unlocks a viable, dictionary-free path toward universal speech recognition for low-resource languages.