Skip to content

A Family Affair: Counterintuitive transfer effects in ASR for endangered languages

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/FormosanBank/A-Family-Affair-ASR
Area: Multimodal VLM
Keywords: Endangered Languages, Automatic Speech Recognition, Cross-lingual Transfer, Phylogenetic Relatedness, Two-stage Fine-tuning

TL;DR

Investigating two-stage cross-lingual transfer for seven endangered Austronesian (Formosan) languages under strict phone-count-matched controls reveals the counterintuitive finding that unrelated donor languages (e.g., English, Swahili, Hungarian) consistently yield larger Word Error Rate (WER) reductions than closely related sister languages, and even outperform nearly doubling target in-language training data.

Background & Motivation

Developing reliable automatic speech recognition (ASR) systems for endangered and low-resource languages is essential for cultural documentation and linguistic revitalization. Contemporary methodologies predominantly fine-tune massive multilingual speech foundation models, such as XLSR-53 or wav2vec 2.0, on whatever small quantity of transcribed audio is available in the target under-resourced language. However, because typical endangered languages possess exceedingly few surviving native speakers and even fewer individuals possessing the technical bandwidth and time to transcribe audio, available datasets are frequently constrained to five or ten hours. Under such minimal training regimes, word error rates (WER) and character error rates (CER) remain stubbornly elevated despite the representational depth of foundation models.

A longstanding and near-ubiquitous rule of thumb across low-resource NLP and speech processing is that when in-language data is scarce, practitioners should augment training sets with data drawn from more resourced languages belonging to the exact same phylogenetic language family. Yet, this practice has rarely been rigorously tested under modern Transformer-based multilingual speech paradigms. Endangered languages often belong to language families that are themselves globally under-resourced; pursuing related language data is therefore technically and logistically costly, while there is remarkably little empirical validation demonstrating that phylogenetic kinship provides superior transfer efficacy compared to readily available unrelated languages or direct in-language expansion.

Furthermore, empirical comparisons in cross-lingual speech transfer are chronically prone to confounding artifacts. These include severe discrepancies in speaking ratesโ€”such as educational scripted recordings spoken at unusually slow paces versus standard conversational natural speechโ€”as well as variations in acoustic recording quality, subword token overlap, and phonemic coverage. Without holding phone-level information volume strictly constant across corpora, assertions regarding phylogenetic benefits remain scientifically unsubstantiated. Core idea: by establishing a rigorously phone-count-normalized two-stage fine-tuning protocol across seven endangered Formosan target languages and eight donor languages, this work uncovers a systematic counterintuitive transfer effect where unrelated donor languages consistently outperform closely related languages in reducing ASR error rates, demonstrating that modern acoustic foundation models benefit more from broad acoustic-phonetic diversity than phylogenetic closeness.

Method

Overall Architecture

The experimental framework establishes an end-to-end, controlled two-stage cross-lingual transfer learning pipeline. The workflow operates across three interconnected phases: first, raw speech corpora from endangered target languages and prospective donor languages are aligned via a universal phone recognizer to eliminate speech rate discrepancies, yielding normalized ~222K phone training partitions; second, an XLSR-53 foundation acoustic encoder undergoes stage-one fine-tuning on a designated donor language using a unified character vocabulary with a Connectionist Temporal Classification (CTC) loss without freezing speech representations; third, the resulting donor checkpoints are adapted in stage-two fine-tuning onto target-language small partitions (~100K phones), followed by pyctcdecode decoding constrained by target-language trigram word language models and statistical mixed-effects regression.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multilingual Speech & Transcripts<br/>7 Formosan Targets + 4 Donor Languages"] --> B["Phone-Count-Based Corpus Normalization<br/>Allosaurus phone extraction to ~222K phones"]
    B --> C["Donor Language Stage-One Fine-Tuning<br/>XLSR-53 + uniform char vocab + CTC training"]
    C --> D["Target Endangered Language Stage-Two Adaptation<br/>Warm-start initialization on 5h target set"]
    D --> E["Covariate-Controlled Regression Analysis<br/>Controlling for relatedness / ATDS / subwords / NISQA"]

Key Designs

1. Phone-Count-Based Corpus Normalization: Eliminating Speaking Rate Confounders

Standard cross-lingual transfer studies typically match dataset size using raw audio duration (e.g., 10 hours), but this metric introduces profound systematic bias when applied to endangered language corpora. The Taiwanese Indigenous language ePark corpus was developed as an educational resource, where native speakers enunciate at an artificially slow speaking rate, whereas non-Formosan corpora (e.g., TED-LIUM 3 English or VoxPopuli Hungarian) exhibit natural presentation-style speech rates containing vastly more phonetic content per hour. To establish strict equivalence, the authors reject raw duration and raw character counts, instead employing the pretrained universal phone recognizer Allosaurus to transcribe all audio into phonetic sequences. Setting the 10-hour Bunun corpus as the standard threshold (~222K phones), every donor dataset (Formosan sister languages and external donors alike) is randomly sampled until reaching exactly ~222K phones. Target small baselines are similarly standardized to ~100K phones (roughly 5 hours) with an 80/20 train/test split, ensuring that observed transfer benefits reflect genuine linguistic dynamics rather than disparities in speech volume.

2. Donor Language Stage-One Fine-Tuning: Structuring Dense Acoustic Latent Geometry

To examine whether intermediate cross-lingual training establishes a superior representational initialization for cold-start endangered languages, the pipeline implements stage-one transfer fine-tuning. Building on the 53-language self-supervised wav2vec 2.0 checkpoint (XLSR-53), a linear projection layer is appended for CTC loss calculation over a unified character set. The full architectureโ€”including convolutional feature encoders and Transformer layersโ€”is fine-tuned for up to 30 epochs (with all models converging well before epoch 30) across each of the eight donor corpora: four closely related Formosan languages (Rukai, Puyuma, Paiwan, Bunun), one distantly related Austronesian language (Tagalog), and three unrelated languages (English, Swahili, Hungarian). This stage adapts the universal acoustic manifold into an operational phoneme-discriminative space.

3. Target Endangered Language Stage-Two Adaptation: In-Language Decoding and Triphone Language Modeling

In the second fine-tuning stage, the converged donor model weights serve as warm-start initializations for training on the target endangered language's small (~100K phone) dataset. Training runs under identical hyperparameter conditions until convergence. During inference, greedy CTC decoding is augmented with pyctcdecode using a trigram word-level language model constructed via KenLM on the target training transcripts, preventing phonemic drift and aligning output sequences with target vocabulary semantics for exact WER and CER scoring.

4. Covariate-Controlled Regression Analysis: Disentangling Kinship from Acoustic Quality

To ensure that empirical improvements are not artifacts of acoustic channel attributes or vocabulary overlaps, the paper implements a linear mixed-effects regression framework with random intercepts per target language. The model simultaneously evaluates the predictive power of: 1. Phylogenetic Relatedness: Formosan related, Tagalog distantly related, and unrelated language families; 2. Acoustic Token Distribution Similarity (ATDS): Acoustic-phonetic sequence overlap derived from XLSR-53 latent representations; 3. Subword Overlap: Token overlap computed via the language-agnostic multilingual Whisper BPE tokenizer; 4. Audio Quality (NISQA): Non-intrusive objective speech quality metric evaluated via the VERSA toolkit across 200 randomly sampled utterances per corpus. This multi-variable modeling formally isolates the net effect of linguistic phylogenetic kinship from recording settings and tokenization biases.

Benchmark & Evaluation Protocol

The benchmark evaluates seven Formosan target languages: Amis, Atayal, Bunun, Paiwan, Puyuma, Rukai, and Seediq. Each target language is evaluated under two internal controls: the Small baseline (5 hours, ~100K phones) and the Large baseline (10 hours, ~220K phones), compared against two-stage fine-tuning across all eight donor languages.

Key Experimental Results

Main Results: Word Error Rate (WER %) Comparison

The table below presents the full Word Error Rate (WER %) evaluation across the seven target Formosan languages for both in-language baselines and all two-stage transfer conditions (bold denotes the top performance per target row):

Target Language Small (5h) Large (10h) Rukai Donor Puyuma Donor Paiwan Donor Bunun Donor Tagalog (Dist.) English (Unrel.) Hungarian (Unrel.) Swahili (Unrel.)
Rukai 54.94 35.92 โ€“ 48.30 42.83 44.85 43.40 40.67 50.80 40.49
Puyuma 32.36 16.33 22.20 โ€“ 15.30 16.71 14.81 13.08 21.64 13.40
Paiwan 44.36 28.82 38.78 36.78 โ€“ 33.09 30.59 27.91 38.98 28.72
Bunun 48.85 32.83 33.32 33.05 29.67 โ€“ 32.89 24.89 33.45 27.33
Amis 37.48 21.38 23.05 21.90 16.59 18.20 17.85 16.17 21.29 16.46
Seediq 35.44 32.07 21.99 20.81 15.70 15.53 14.50 13.38 19.64 13.28
Atayal 44.70 21.01 25.11 24.46 18.96 18.83 17.49 16.36 24.50 18.28

Character Error Rate (CER %) Comparison

The following table provides character error rate (CER %) figures across all configurations, confirming consistent acoustic gains across phonemic boundaries:

Target Language Small Base Large Base Best Formosan Donor Tagalog Donor English Donor Hungarian Donor Swahili Donor
Rukai 11.87 5.90 7.52 (Bunun) 21.50 6.79 9.27 6.84
Puyuma 8.77 4.53 3.91 (Paiwan) 10.94 3.62 4.99 3.71
Paiwan 10.87 4.09 5.53 (Puyuma) 5.35 4.13 6.27 4.56
Bunun 11.37 6.82 4.82 (Paiwan) 12.63 3.81 5.06 4.01
Amis 9.67 8.54 3.17 (Paiwan) 7.32 2.98 3.81 3.20
Seediq 30.06 13.94 6.28 (Bunun) 8.18 6.06 6.98 5.74
Atayal 20.59 10.61 6.15 (Paiwan) 9.16 5.65 7.36 6.01

Key Findings

  • The Counterintuitive Relatedness Penalty: While two-stage fine-tuning universally improves upon the Small baseline across all target languages, unrelated donor languages consistently outperform the best related Formosan donor. A linear mixed-effects regression confirms that Formosan sister donors yield significantly less WER reduction than unrelated donors (\(\beta = -0.06, 95\%\ \text{CI}\ [-0.11, -0.01], p = 0.010\)). Distantly related Tagalog showed no significant difference from unrelated donors (\(\beta = +0.01, p = 0.78\)).
  • Covariates Amplify the Unrelated Advantage: When acoustic similarity (ATDS), subword overlap, and NISQA recording quality are added as regression covariates, the Formosan penalty intensifies from \(-0.06\) to \(-0.20\) (\(p < 0.001\)), while Tagalog displays a \(+0.35\) advantage (\(p < 0.001\)). Subword overlap (\(\beta = +0.04, p = 0.001\)) and NISQA (\(\beta = +0.16, p < 0.001\)) both positively correlate with WER reduction, demonstrating that the deficit of related donors cannot be explained away by surface acoustic or textual dissimilarities.
  • Unrelated Transfer Exceeds Doubling In-Language Data: In six of the seven endangered target languages, two-stage fine-tuning from an unrelated donor (notably English or Swahili) achieves lower WER than training on double the target in-language data (Large baseline). A paired Wilcoxon signed-rank test establishes that best-donor WERs are statistically superior to Large baseline WERs (\(W = 25, p < 0.05\)).
  • Generalization Beyond Foundation Pretraining: Even though English, Swahili, and Tagalog were included in XLSR-53 pre-training data, Hungarian was entirely absent from XLSR-53 pre-training. Nevertheless, Hungarian proves to be an effective donor, surpassing three out of four related Formosan donors in WER reduction, confirming that the transfer benefit is not merely an artifact of foundation data memorization.

Highlights & Insights

  • Challenging a Foundational Paradigm: Overturns decades of accepted low-resource ASR wisdom prescribing related-language data collection, demonstrating that acoustic foundation models gain more functional discriminability from diverse, high-resource unrelated speech.
  • Phone-Count Normalization Standard: Exposes that duration-based multi-corpus evaluation heavily distorts low-resource speech research due to extreme speaking rate variance, providing a clean Allosaurus-based phonemic normalization paradigm.
  • Actionable Guidance for Field Linguistics: For endangered language communities with minimal transcription resources, leveraging freely available high-resource corpora (such as English or Swahili) in a two-stage setup provides higher accuracy gains than laboriously collecting additional in-language speech or seeking scarce sister-language audio.

Limitations & Future Work

  • Foundation Architecture Scope: The study focuses exclusively on XLSR-53 because alternative architectures like HuBERT exhibited inferior baselines and Whisper demonstrated high vulnerability to multi-stage out-of-domain degradation. Future research must probe whether newer open multilingual architectures mirror these dynamics.
  • Speaker Diversity Confounding: While speaker counts were available for Hungarian, other corpora lacked ground-truth speaker diarization labels, leaving the exact quantitative contribution of speaker diversity across donor corpora to be fully disentangled.
  • Phylogenetic Generalization: Experiments centered on the Formosan sub-family of Austronesian languages; validating this counterintuitive phenomenon across Indigenous American and African language families remains an important next step.
  • vs. Coto-Solano et al. (2024): While prior work reported benefits from fine-tuning related Chibchan language models, it did not benchmark against phone-matched unrelated donor baselines; this study demonstrates that unrelated transfer actually provides superior margins.
  • vs. Jimerson et al. (2023): Whereas earlier findings highlighted the unpredictable performance of various model architectures in extreme low-resource regimes, this paper demonstrates a robust, architecture-consistent transfer protocol that reliably outperforms monolingual baselines.

Rating

  • Novelty: โญโญโญโญโญ Strikingly overturns standard assumptions in endangered language ASR with rigorous empirical counter-evidence.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive phone-count normalization, dual-stage training across 7 targets and 8 donors, and exhaustive mixed-effects regression.
  • Writing Quality: โญโญโญโญโญ Cohesive, transparent, mathematically principled narrative that proceeds logically from motivation to statistical validation.
  • Value: โญโญโญโญโญ Provides profound practical and computational cost savings for global language documentation and field linguistics.