Using Prosody to Predict Syntactic Structure¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/aatlantise/prosody-syntax-interface
Area: LLM Reasoning
Keywords: Syntax-Prosody Interface, Mutual Information, Sequence-to-Sequence Modeling, Prosodic Bootstrapping, Error-Correcting Code
TL;DR¶
By formulating the syntax-prosody interface as sequence-level mutual information estimation using an encoder-decoder architecture over read (LibriTTS) and spontaneous (CANDOR) speech, this paper reveals that acoustic features (word duration, pause) reduce syntactic uncertainty by up to 10.2%, predominantly marking phrase boundaries rather than syntactic categories.
Background & Motivation¶
Prosody, commonly referred to as the melodic contour and timing rhythm of human speech, carries vital communicative cues that have long been suspected to reflect underlying syntactic representations. Developmental psycholinguistics has demonstrated that infants rely on prosodic boundaries to scaffold syntactic structure acquisition before acquiring vocabulary, speakers systematically produce pre-boundary lengthening to signal major constituent boundaries, and listeners utilize tonal and pause patterns to resolve syntactic ambiguities during real-time comprehension. However, the exact nature and degree of informational coupling at the syntax-prosody interface remain intensely debated in theoretical linguistics: Direct Reference Theory (DRT) posits that prosodic phrasing directly reflects syntactic constituency without intervening phonological structure; Indirect Reference Theory (IRT) argues that syntax and phonology represent autonomous modules that interact through mediating alignment constraints subject to phonological well-formedness; and Prosody-Driven Syntax (PDS) suggests that prosody exerts an active, bidirectional influence on syntactic operations, particularly in spontaneous discourse.
Despite decades of scholarly inquiry, empirical validation of these conflicting theories has faced severe methodological bottlenecks. Traditional empirical investigations have predominantly relied on controlled phonetic experiments conducted in laboratory settings with small, artificial sets of isolated sentences, which fail to capture the distribution and messiness of natural conversational speech. On the other hand, corpus-level studies have historically depended on rudimentary statistical classifiers or made oversimplified assumptions, such as omitting lexical and surrounding syntactic contexts, thereby conflating lexical predictability with true acoustic prosodic contributions. Crucially, because acoustic dimensions such as pitch and duration are continuous-valued, previous computational models estimating continuous entropy lacked theoretical lower bounds, rendering it impossible to quantify the absolute bits of structural uncertainty that prosody resolves.
To resolve these longstanding empirical challenges, this work introduces a rigorous, structure-agnostic, and context-aware information-theoretic framework to measure the mutual information between prosody and syntax over large-scale speech corpora. By inverting the estimation pipeline to predict discrete linearized constituent trees conditioned on continuous acoustic features and textual context, the approach establishes an upper-bounded cross-entropy estimation protocol that permits valid cross-genre comparisons across planned read audiobooks and unscripted naturalistic conversations. Core idea: Formulating the syntax-prosody relationship as mutual information over discrete syntactic sequences, deploying a lightweight sinusoidal prosody encoder within a T5 sequence-to-sequence pipeline to rigorously quantify the uncertainty reduction provided by duration and pauses.
Method¶
Overall Architecture¶
The proposed framework treats text word tokens \(W\), linearized constituency parse tokens \(S\), and continuous prosodic unit features \(P\) (specifically word duration \(P_{\text{dur}}\) and inter-word pauses \(P_{\text{pause}}\)) as random variables sampled from a joint data-generating distribution. Under the good mixed-pair assumption, the syntactic information content in prosody is operationalized as mutual information \(I(P, S) = H(S) - H(S \mid P)\), while the incremental information carried by prosody over segmental words is measured as conditional mutual information \(I(P, S \mid W) = H(S \mid W) - H(S \mid W, P)\). To reliably estimate these quantities on sentence-level sequences without length normalization artifacts, the pipeline employs multimodal encoder-decoder language models trained via teacher forcing to minimize cross-entropy, which serves as a tight upper bound on true sequence-level entropy.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Speech & Text Input Streams<br/>MFA Aligned Durations/Pauses + Transcripts"] --> B["Multi-Granular Parse Linearization<br/>Full Constituent Trees vs Bracket Hierarchies"]
B --> C["Dual Sinusoidal Encoding & Cross-Modal Fusion<br/>Magnitude-Position Concatenation + Cross-Attention"]
C --> D["Cross-Entropy & Mutual Information Estimation<br/>Autoregressive Upper-Bound over Unimodal/Multimodal Models"]
D --> E["Cross-Genre & Disfluency Controlled Testing<br/>LibriTTS vs CANDOR + Length-Matched Wilcoxon Tests"]
Key Designs¶
1. Multi-Granular Parse Linearization: Isolating Structural Boundaries from Syntactic Category Labels
Existing syntactic probing literature often conflates syntactic topology (where phrase breaks occur) with phrasal identity (the lexical and category labels such as NP, VP, or SBAR). To determine precisely which components of syntactic structure prosody encodes, this framework introduces two distinct levels of linearized parse granularity based on the Penn Treebank specification: a "Full Parse" representation generated by the Stanza neural parser (silver annotations, with punctuation stripped) comprising complete bracketings and POS/phrasal category symbols, and a "Bracket-Only" representation where all category labels are discarded, leaving only opening and closing parentheses that dictate hierarchical constituent nesting. This systematic decoupling provides an exact mathematical lens to isolate whether durational cues inform the structural segmentation of speech or the higher-order grammatical categories of the constituents.
2. Dual Sinusoidal Encoding & Cross-Modal Fusion: Bridging Continuous Acoustic Signals to Discrete Language Embeddings
Continuous prosodic measurements—specifically millisecond word durations normalized by CELEX syllable counts and inter-word pause lengths binned into 10 ms increments—cannot be natively ingested by discrete Transformer tokenizers. To prevent overfitting on small continuous parameter spaces while preserving metric magnitude and temporal sequence order, the authors design a shallow, dedicated prosody encoder (2 Transformer layers, 2 attention heads, hidden dimension \(D=8\)) that receives concatenated sinusoidal magnitude encodings (\(\text{ME}\)) and position encodings (\(\text{PE}\)): $$ \text{ME}(\text{mag}, d) = \begin{cases} \sin\left(\frac{\text{mag}}{20000^{2i/D}}\right) & \text{if } d = 2i \ \cos\left(\frac{\text{mag}}{20000^{2i/D}}\right) & \text{if } d = 2i+1 \end{cases}, \quad \text{PE}(\text{pos}, d) = \begin{cases} \sin\left(\frac{\text{pos}}{10000^{2i/D}}\right) & \text{if } d = 2i \ \cos\left(\frac{\text{pos}}{10000^{2i/D}}\right) & \text{if } d = 2i+1 \end{cases} $$ The resulting acoustic representations are projected to the dimension of a pre-trained T5-base encoder. For joint conditioning \(p_\theta(s \mid w, p)\), a cross-modal cross-attention layer is inserted with text representations serving as Query and prosody representations as Key and Value, stabilized by layer normalization and dropout before being passed to the decoder stack.
3. Cross-Entropy & Mutual Information Estimation: Bounded Information-Theoretic Probing via Sequence Prediction
Prior research evaluating the mutual information of prosody attempted to predict continuous acoustic targets from text representations, which led to unbounded differential entropy estimates that precluded meaningful cross-model normalization. This paper circumvents the issue by inverting the objective: predicting discrete linearized parse tokens \(s \in \Upsilon^*\) from conditioning inputs. By invoking the property that cross-entropy \(H_\theta(S) = - \mathbb{E}_{s \sim p}[\log p_\theta(s)]\) strictly upper-bounds true Shannon entropy \(H(S)\) by the non-negativity of Kullback-Leibler divergence \(D_{\text{KL}}(p \parallel p_\theta)\), the mutual information \(I(P, S) = H(S) - H(S \mid P)\) and the uncertainty coefficient \(U = I(P, S) / H(S)\) quantify the exact percentage reduction in parse uncertainty delivered by prosodic features.
4. Cross-Genre & Disfluency Controlled Testing: Disentangling Speech Style and Utterance Length Confounders
Spontaneous conversational speech differs intrinsically from read speech in sentence length, structural regularity, and the presence of speech disfluencies (filled pauses like "um" and "uh", repetitions, and false starts). To ensure that observed differences between planned speech (LibriTTS, 44k sentences, \(\bar{l}=17.5\) words) and unscripted dialogue (CANDOR, 298k sentences, \(\bar{l}=13.2\) words) are not artifacts of utterance length distributions or disfluent noise, the framework introduces: (1) Wilcoxon signed-rank paired statistical testing across cross-validation folds; (2) a length-matched sampling protocol constraining sentence lengths to identical bins (\(5 \le l \le 25\) words); and (3) an Ordinary Least Squares (OLS) regression analysis testing the incremental explanatory power of binary disfluency indicators (\(\Delta R^2 < 0.01\)).
Loss & Training¶
All models are optimized using sequence-level cross-entropy loss under teacher forcing. The vocabulary is augmented with 66 specialized parse and POS tokens, which receive randomly initialized embeddings strictly utilized on the decoding side. Training utilizes AdamW with an initial learning rate of 3e-4, scheduled via cosine annealing down to zero, with a maximum sequence length of 256 and early stopping patience of 3 epochs. Cross-validation employs 10 folds for LibriTTS (90:10 train/test splits, taking approximately 4 GPU-days per run on an NVIDIA Tesla L4) and 5 folds for CANDOR (80:20 train/test splits, taking approximately 2 GPU-days on an NVIDIA H100).
Key Experimental Results¶
Main Results¶
The primary evaluation reports sentence-level syntactic entropy estimates \(H\) (in nats per sequence) across input conditioning modalities, parse granularities, and speech domains.
| Genre | Parse Type | \(H(S)\) Prior | \(H(S \mid P_{\text{pause}})\) Pause | \(H(S \mid P_{\text{dur}})\) Duration | \(H(S \mid W)\) Text | \(H(S \mid W, P_{\text{pause}})\) Text+Pause | \(H(S \mid W, P_{\text{dur}})\) Text+Duration |
|---|---|---|---|---|---|---|---|
| Spontaneous (CANDOR) | Brackets | 13.90 | 12.70 | 12.40 | 2.83 | 3.03 | 3.06 |
| Spontaneous (CANDOR) | Full Parse | 37.20 | 35.10 | 33.70 | 5.25 | 5.27 | 5.30 |
| Planned (LibriTTS) | Brackets | 16.90 | 16.10 | 16.00 | 5.76 | 6.97 | 7.13 |
| Planned (LibriTTS) | Full Parse | 48.20 | 46.20 | 46.60 | 12.80 | 13.60 | 13.50 |
Mutual Information and Uncertainty Reduction Analysis¶
The table below details the estimated mutual information \(I\) (in nats) and the corresponding uncertainty coefficient \(U = I / H(S)\) for prosodic features and lexical text. All positive mutual information estimates are statistically significant under the Wilcoxon signed-rank test (\(p < 0.001\)).
| Genre | Parse Type | \(I(S, P_{\text{pause}})\) | \(U(P_{\text{pause}})\) | \(I(S, P_{\text{dur}})\) | \(U(P_{\text{dur}})\) | \(I(S, W)\) Text | \(U(W)\) Text | \(I(S, P_{\text{pause}} \mid W)\) | \(I(S, P_{\text{dur}} \mid W)\) |
|---|---|---|---|---|---|---|---|---|---|
| Spontaneous | Brackets | 1.20 nats | 8.64% | 1.42 nats | 10.20% | 11.00 nats | 79.60% | -0.20 nats | -0.24 nats |
| Spontaneous | Full Parse | 2.05 nats | 5.52% | 3.45 nats | 9.27% | 31.90 nats | 85.90% | -0.02 nats | -0.06 nats |
| Planned | Brackets | 0.75 nats | 4.44% | 0.84 nats | 4.99% | 11.10 nats | 65.90% | -1.21 nats | -1.37 nats |
| Planned | Full Parse | 2.06 nats | 4.27% | 1.65 nats | 3.43% | 35.40 nats | 73.50% | -0.89 nats | -0.73 nats |
Length-Controlled and Disfluency Robustness Analysis¶
When sampling identical sentence-length distributions (\(5 \le l \le 25\)) to eliminate sentence length discrepancies between domains, spontaneous speech retains a statistically significant advantage in syntactic information content: - Duration Mutual Information: CANDOR achieves 3.59 nats versus 1.55 nats in LibriTTS (\(p < 0.001\)); - Pause Mutual Information: CANDOR achieves 2.13 nats versus 1.94 nats in LibriTTS (\(p < 10^{-17}\)). In analyzing the 19.8% of CANDOR sentences exhibiting medial disfluencies (repetitions and filled pauses), uncontrolled estimates indicated higher raw mutual information in disfluent subsets (duration: 3.91 nats vs. fluent 3.31 nats). However, controlling for sentence length (\(N=58\text{k}\) matched pairs, \(\bar{l}=17.28\) words) shrank this discrepancy from 40% to within 7.5%, with OLS regression confirming that length, rather than disfluency itself, drives cross-subset variation (\(\Delta R^2 < 0.01\)).
Key Findings¶
- Prosody carries statistically significant syntactic information: In unscripted spontaneous dialogue, word duration reduces structural uncertainty by up to 10.2% and inter-word pauses by 8.64%, confirming that speech acoustics encode meaningful syntactic structure in the absence of lexical content.
- Syntactic cues predominantly mark phrase boundaries: Phrase brackets account for approximately 50% of the syntactic information in duration (1.42 out of 3.45 nats in CANDOR) and 60% of the information in pauses (1.20 out of 2.05 nats in CANDOR), demonstrating that durational timing primarily acts as a structural boundary indicator rather than a phrasal category discriminator.
- Redundancy with lexical content: When conditioned on known word identities, \(I(S, P \mid W)\) reveals no statistically significant positive information gain, demonstrating that the syntactic information provided by duration and pauses is redundant with the grammatical constraints conveyed by words.
- Cross-genre divergence validates Prosody-Driven Syntax: Spontaneous unscripted conversations display systematically higher uncertainty reduction (\(U\)) from prosody than planned read speech (higher by roughly 4% to 6 percentage points), supporting the theoretical premise that online language production dynamically coordinates prosodic and syntactic structures.
Highlights & Insights¶
- Inverted information-theoretic probing: By reformulating mutual information estimation from continuous acoustic prediction to discrete sequence-to-sequence tree parsing, the authors circumvented the unbounded differential entropy trap, establishing an elegant, mathematically grounded methodology for linguistic probing.
- Decisive empirical adjudication of linguistic theories: The findings provide direct quantitative evidence distinguishing longstanding grammar theories: rejecting Direct Reference Theory's prediction of near-complete isomorphism while strongly corroborating Indirect Reference Theory (moderate ~10% uncertainty reduction) and Prosody-Driven Syntax (heightened spontaneous coupling).
- Prosody as an error-correcting channel: While prosodic timing is redundant in pristine textual transcripts, its presence in acoustic speech acts as an error-correcting code, offering an orthogonal biological channel for syntactic parsing when acoustic degradation, background noise, or unfamiliar vocabulary impede lexical recognition.
Limitations & Future Work¶
- Unexplored acoustic dimensions and language diversity: The investigation is restricted to English temporal features (duration and pauses), omitting continuous fundamental frequency (\(F_0\)/pitch), spectral energy, and composite prominence, which are known to be critical for tonal and pitch-accent languages.
- Asymmetric cross-modal representation capacity: The pre-trained T5 text encoder provides an exceptionally strong representation of \(W\), whereas the small, randomly initialized prosody encoder and cross-attention fusion layer have lower capacity, causing slightly deflated joint entropy upper bounds and resulting in slightly negative empirical \(I(S, P \mid W)\) estimates.
- Silver parse noise in conversational transcripts: The reliance on Stanza silver parses trained predominantly on written data introduces potential annotation noise, particularly for fragmented or ungrammatical conversational turns.
Related Work & Insights¶
- vs. Wolf et al. (2023) & Wilcox et al. (2025): While prior studies explored mutual information between speech acoustics and segmental tokens, their reliance on continuous feature distributions constrained entropy estimation; the present work resolves this by targeting discrete syntactic sequences.
- vs. Yadavalli et al. (2025): Whereas Yadavalli et al. demonstrated the non-redundant utility of prosody for affective classification tasks (emotion, sarcasm), this study extends the information-theoretic framework to formal grammar and demonstrates high lexical redundancy in syntactic constituency.
- vs. Classical prosodic boundary parsing (Kahn et al., 2005; Cho, 2022): Previous computational approaches utilized prosody as hand-engineered features for improving conversational speech recognizers or punctuation restorers; this work abstracts away from heuristic parsing performance to uncover fundamental theoretical dependencies.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering adaptation of sequence-level mutual information to the syntax-prosody interface with a discrete parsing target.]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive cross-genre evaluation across 342k sentences, rigorous cross-validation, and principled post-hoc controls for length and disfluency.]
- Writing Quality: ⭐⭐⭐⭐⭐ [Flawlessly structured, combining dense linguistic theoretical rigor with modern machine learning exposition.]
- Value: ⭐⭐⭐⭐⭐ [Offers foundational empirical grounding for psycholinguistics, prosodic bootstrapping, and the design of unified speech-language models.]