Entropic properties of languages in phonetic versus orthographic alphabets¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://doi.org/10.5281/zenodo.22133437
Area: LLM (Other)
Keywords: Cross-modal entropy, Phonetics vs. Orthography, Byte-Pair Encoding (BPE), Information theory, Cross-linguistic analysis
TL;DR¶
Using the DoReCo natural speech corpus spanning 42 typologically diverse languages across 25 families, this paper computes continuous Shannon entropy trajectories across word, subword (BPE), and character/phone granularities, demonstrating a systematic modality convergence between orthographic and phonetic encodings with the smallest gap observed at the subword level.
Background & Motivation¶
Computational linguistics and modern natural language processing predominantly rely on written text for model training, tokenization, and theoretical evaluation. This persistent orthographic bias tacitly assumes that written representations serve as the primary and sufficient carrier of linguistic structure. From an evolutionary and linguistic perspective, however, writing systems represent spoken language only indirectly and imperfectly. Whether spoken (phonetic) and written (orthographic) modalities share fundamental information-theoretic properties such as Shannon entropy remains an open empirical question that has lacked large-scale cross-linguistic validation.
Directly comparing information-theoretic metrics across modalities introduces substantial methodological hurdles. Strictly aligned multimodal corpora featuring both natural speech and orthographic transcriptions across diverse language families are extraordinarily scarce, risking spurious entropy gaps driven by mismatched genres or registers. Furthermore, cross-linguistic variations in orthographic depth, morphological typology, and phonemic inventory make fixed-length sequence slicing or endpoint-only inventory comparisons prone to truncation bias and blind to the continuous dynamics of discourse. Prior studies either focused on a single Indo-European language or omitted multi-granularity (character, subword, word) evaluations.
To bridge this gap, this paper leverages the Cross-Linguistic Data Format (CLDF) release of the DoReCo corpus, covering 42 typologically diverse languages across 25 genealogical families. By tracking continuous cumulative Shannon entropy trajectories along the natural discourse stream, the authors systematically examine the information density and convergence dynamics between orthographic (UTF-8) and phonetic (X-SAMPA) representations across words, subwords, and characters. Core idea: by constructing continuous Shannon entropy trajectories across 42 languages, this study demonstrates a systematic tendency toward modality convergence across word, subword, and character granularities, reaching a minimal modality gap at the BPE subword level (averaging just 0.022 bits difference), while uncovering how language-specific phonological inventories and orthographic depth drive granular divergence.
Method¶
Overall Architecture¶
The analytical pipeline consists of corpus filtering and alignment, dual-modality three-granularity sequence construction, unsupervised subword tokenization via Byte-Pair Encoding (BPE), and continuous cumulative Shannon entropy estimation. The input comprises aligned audio transcriptions from the DoReCo CLDF dataset across 42 languages, and the output yields six continuous entropy trajectories per language alongside cross-linguistic convergence statistics. The workflow unfolds in three main phases: first, raw transcription annotations, pause tokens, and words lacking phone transcriptions are stripped while strictly preserving word ID sequence ordering; second, parallel sequences are generated across written (UTF-8) and spoken (X-SAMPA) streams at word, subword, and character/phone levels; finally, dynamic cumulative unigram Shannon entropy is estimated across the token stream using the maximum likelihood plug-in estimator.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["DoReCo CLDF Corpus<br/>42 languages across 25 families"] --> B["Multimodal Stream Alignment & Cleaning<br/>Strip annotation tags & preserve word ID order"]
B --> C["Dual-Modality Three-Granularity Sequences<br/>Written UTF-8 vs. Spoken X-SAMPA"]
C --> D["Unsupervised Cross-Modal BPE Tokenization<br/>Whitespace-free concatenation on phone stream"]
D --> E["Continuous Shannon Entropy Trajectory Estimation<br/>Cumulative prefix maximum-likelihood distribution"]
E --> F["Cross-Modal & Cross-Linguistic Convergence Analysis<br/>Compute modality gap and typological variance"]
Key Designs¶
1. Multimodal Stream Alignment & Cleaning: Eliminating transcription artifacts while preserving natural discourse ordering
Cross-modal entropy calculations are acutely vulnerable to annotation artifacts and sequence discontinuities. To ensure uncorrupted comparisons across DoReCo languages, the cleaning pipeline applies three strict filters: first, all annotation tags (e.g., <<tag>content>), pause tokens (<p:>, <p>, <sil>), and placeholder markers (****, [INT]) are uniformly removed across all encoding streams; second, tokens are sorted strictly by their sequential word ID to preserve natural conversational chronology within recording sessions and keep speaker contributions contiguous; third, words lacking corresponding phone transcriptions in DoReCo are discarded simultaneously from both orthographic and phonetic streams to ensure strict paired comparability.
2. Dual-Modality Three-Granularity Sequences: Decoupling orthographic and phonological representation spaces
To conduct rigorous cross-modal comparisons, the method constructs six parallel representations spanning three distinct linguistic tiers: at the word level, orthographic words are contrasted against phone words (where the complete phonetic string of each word is treated as a single token); at the character level, individual UTF-8 characters are contrasted against individual X-SAMPA phones; at the subword level, data-driven units are derived via BPE. This multi-tiered mapping exposes structural encoding asymmetries across languages—for example, Bora orthography marks tone with accents (<í>) whereas SAMPA omits tone and only marks length ([i:]), and Sanzhi Dargwa writes ejectives with a single character (<cij>) but transcribes them with multi-character digraphs in SAMPA ([ts_>]).
3. Unsupervised Cross-Modal BPE Tokenization: Capturing phonological and orthographic compression boundaries
To evaluate whether algorithmic subword segmentation operates consistently across modalities, language-specific BPE tokenizers are trained independently on orthographic character streams and phonetic streams. For the phonetic stream, phones are concatenated without whitespace, empowering the merge algorithm to learn natural morphological and phonological boundaries directly from co-occurrence statistics, naturally fusing base phones with diacritics and length markers. The default merge budget is set to 200 merges, complemented by sensitivity analyses at 100 and 400 merges to assess the stability of cross-modal compression.
4. Continuous Shannon Entropy Trajectory Estimation: Tracking dynamic information rate stabilization
Rather than calculating static entropy solely on full corpora endpoints, the methodology computes continuous entropy trajectories across the token stream. At each position \(i\), the empirical probability distribution is calculated over the cumulative prefix from position 1 to \(i\): $\(\hat{p}(x_j) = \frac{f_j}{\sum_{k=1}^{|\mathcal{X}|} f_k}\)$ Shannon entropy is then estimated via the plug-in formula: $\(H(X) = -\sum_{x \in \mathcal{X}} p(x) \log_2 p(x)\)$ Evaluating continuous trajectories demonstrates that character-level and subword-level entropies plateau rapidly within the first 500–1000 tokens due to smaller inventory sizes, while word-level entropy grows continuously and decelerates as lexical items recur, ensuring findings are robust to corpus size variations.
Loss & Training¶
This study focuses on empirical information-theoretic estimation and unsupervised BPE tokenization. Language-specific BPE models are trained greedily using frequency-based pair merges on single-modality token streams with 200 merges (alongside 100 and 400 merges for sensitivity verification). Entropy estimation relies on the standard maximum likelihood plug-in estimator without smoothing, avoiding estimation biases or architectural overfitting associated with neural language models on low-resource and small documentary linguistic corpora.
Key Experimental Results¶
Main Results¶
The primary evaluation computes the mean Shannon entropy (Mean \(H\)), cross-linguistic standard deviation (SD), and absolute modality gap (\(\Delta H = |\text{Mean } H_{\text{written}} - \text{Mean } H_{\text{spoken}}|\)) across all 42 DoReCo languages at the final token position.
| Granularity | Encoding | Mean Shannon Entropy \(H\) (bits) | Cross-Linguistic SD | Modality Gap \(\Delta H\) (bits) |
|---|---|---|---|---|
| Word level | Written (Orthographic) | 9.347 | 1.015 | 0.152 |
| Word level | Spoken (Phone words) | 9.195 | 1.054 | - |
| Subword level (BPE 200 merges) | Written (Orthographic) | 5.955 | 0.361 | 0.022 |
| Subword level (BPE 200 merges) | Spoken (Phone subwords) | 5.977 | 0.358 | - |
| Character/Phone level | Written (UTF-8 chars) | 4.314 | 0.360 | 0.139 |
| Character/Phone level | Spoken (SAMPA phones) | 4.453 | 0.289 | - |
Ablation Study¶
A sensitivity analysis assesses the effect of varying BPE vocabulary sizes (100, 200, and 400 merges) on subword entropy and the resulting modality gap.
| BPE Merge Budget | Modality | Mean Shannon Entropy \(H\) (bits) | Cross-Linguistic SD | Modality Gap \(\Delta H\) (bits) |
|---|---|---|---|---|
| 100 merges | Written | 5.207 | 0.257 | 0.153 |
| 100 merges | Spoken | 5.360 | 0.240 | - |
| 200 merges (default) | Written | 5.955 | 0.361 | 0.022 |
| 200 merges (default) | Spoken | 5.977 | 0.358 | - |
| 400 merges | Written | 6.549 | 0.437 | 0.017 |
| 400 merges | Spoken | 6.532 | 0.423 | - |
Key Findings¶
- Tightest Modality Convergence at Subword Level: Across all three granularities, BPE subwords display the smallest average modality gap. At 200 merges, the written and spoken entropy trajectories differ by a mere 0.022 bits (5.955 vs. 5.977 bits), shrinking further to 0.017 bits at 400 merges. This reveals that unsupervised frequency-based subword compression harmonizes information density across speech and text.
- Finer Granularity Sharpens Cross-Linguistic Consistency: Cross-linguistic standard deviation drops steeply from the word level (SD 1.015–1.054) down to the subword (SD 0.358–0.361) and character levels (SD 0.289–0.360), indicating that finer linguistic encodings converge on highly uniform information-theoretic capacities across languages.
- Typologically Conditioned Divergence:
- Complete Convergence: Evenki and Jahai show nearly identical entropy trajectories across all three granularities.
- Character-Level Divergence: Southern English exhibits parity at word and subword levels, but character-level SAMPA phone entropy noticeably exceeds UTF-8 character entropy due to opaque orthography (e.g.,
<ea>mapping to multiple vowels like[I@],[e],[i:]), which BPE merges neutralize early on. - Word/Character-Level Divergence: Bora shows higher written entropy than spoken entropy at word and character levels because Latin orthography marks tonal distinctions that are omitted in phonetic transcription.
- Subword-Level Divergence: Sanzhi Dargwa exhibits a wide modality gap specifically at the subword level, where multi-character representations of its 50+ consonant phonemes (e.g., ejectives like
[ts_>]) produce longer phonetic subword units and higher entropy.
Highlights & Insights¶
- Provides large-scale cross-linguistic empirical evidence that spoken and written language modalities are not informationally disconnected, demonstrating an intrinsic modality-invariance in human communication.
- Discovers that BPE tokenization naturally discovers an optimal cross-modal compression point at 200–400 merges, bringing speech and text representations to an equivalent information density of approximately 5–6 bits without manual linguistic supervision.
- Proposes cross-modal entropy trajectory symmetry as a principled intrinsic benchmark for tokenizer evaluation: divergent entropy trajectories across paired speech and text inputs signal that tokenizers capture modality-specific surface artifacts rather than underlying linguistic structures, which directly informs speech-text multimodal foundation models (e.g., mSLAM).
Limitations & Future Work¶
- Latin Script Constraint: All 42 languages are transcribed using Latin alphabets at the word tier; non-alphabetic writing systems such as logographic scripts (Chinese Hanzi), syllabaries (Devanagari), or abjads (Arabic) may exhibit significantly larger modality discrepancies.
- Transcription Conventions: Minor asymmetries stem from corpus-specific annotation choices (e.g., omitting tonal markers in Bora phonetics), reflecting human transcription limitations rather than intrinsic linguistic divergence.
- Unigram Focus: The analysis relies on unigram Shannon entropy; evaluating conditional entropy, mutual information, and contextual perplexity using neural language models represents an essential direction for future study.
Related Work & Insights¶
- vs. Coupé et al. (2019): Coupé et al. demonstrated that human languages achieve similar information rates in physical speech transmission (~39 bits/s); this work shows that even in the symbolic representation space detached from acoustic duration, orthography and phonetics converge on similar discrete Shannon entropy profiles.
- vs. Gutierrez-Vasques et al. (2021, 2023): Prior work identified a morphological "turning point" at 200–350 BPE merges where cross-linguistic entropy distributions align; this paper demonstrates that this turning point simultaneously serves as a cross-modal convergence point, unifying graphemic and phonetic distributions.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ First large-scale empirical evaluation of continuous Shannon entropy trajectories across spoken and written modalities on 42 diverse languages.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous coverage of 25 language families, systematic three-tier granularity analysis, multi-budget BPE sensitivity checks, and rich morphological case studies.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, rigorous linguistics and information-theoretic formulations, and clear implications for NLP tokenization.
- Value: ⭐⭐⭐⭐☆ Offers foundational empirical insights for speech-text multimodal tokenization, cross-modal transfer, and quantitative linguistics.