Skip to content

Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/naver/tts-friendly-gen
Area: Audio & Speech
Keywords: text-to-speech, preference alignment, speech synthesis, voice assistants, interpretable reward models

TL;DR

This paper frames TTS-friendly text generation as a multi-objective preference alignment problem, leveraging the interpretable Feature-aware Sampling and Tuning (FaST) framework and a comprehensive evaluation suite to make LLMs natively output speakable text without post-processing latency, excelling even under extreme 10-sample data constraints.

Background & Motivation

Current large language models (LLMs) are predominantly optimized on written corpora and standard human conversational preferences. While this produces grammatically sound, detailed, and contextually helpful text, the resulting outputs are routinely ill-suited for downstream acoustic delivery by Text-to-Speech (TTS) synthesizers. In practical voice assistants and spoken dialogue systems, raw LLM outputs frequently contain written conventions such as abbreviations, punctuation symbols, raw URLs, email addresses, and compact numeric formatting (e.g., currency symbols, timestamps, or fractions). When fed into off-the-shelf TTS engines, these surface patterns trigger mispronunciations, unnatural prosodic pauses, or jarring spelling artifacts that degrade user comprehension.

Conventional voice pipelines attempt to remediate this issue by inserting a post-hoc text normalization (TN) module between the LLM and the TTS synthesizer. However, this downstream rewriting approach introduces critical system bottlenecks. Post-processing modules must wait for full clauses or sentences to complete before execution, which substantially increases first-token and end-of-turn latency in real-time voice streaming. Furthermore, external normalization modules duplicate computational inference costs, risk altering or dropping essential factual information, and remain tightly bound to the specific tokenizers and rules of individual TTS systems.

Rather than patching spoken artifacts downstream, text generation should be rendered speech-ready at the source. This work addresses the problem upstream by formalizing speech-readiness as a multi-objective preference alignment task. Core idea: formalize TTS-friendly text generation as a multi-objective preference alignment problem, adopting the interpretable Feature-aware Sampling and Tuning (FaST) framework with linear scalar reward weighting over discovered linguistic features, enabling LLMs to natively generate speech-ready text with minimal latency even under 10-sample low-data regimes.

Method

Overall Architecture

The proposed framework targets the dominant cascade architecture in voice assistants (\(\text{LLM} \to \text{TTS}\)), training the generator to avoid TTS-unfriendly surface forms while preserving response helpfulness and factual correctness. The pipeline comprises four sequential stages: automated feature discovery from preference tuples, frozen LLM-based feature-wise response scoring, parameter-efficient feature weight learning via Bradley-Terry optimization (FaRM), and iterative policy alignment using rejection-sampling fine-tuning (RFT).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Preference Datasets<br/>CORA & Recipe paired samples"] --> B["Feature Discovery<br/>LLM extracts contrasting descriptors"]
    B --> C["Feature-wise Response Scoring<br/>Frozen LLM scoring functions"]
    C --> D["Feature Weight Learning<br/>FaRM linear scalar optimization"]
    D --> E["Sampling & Tuning Alignment<br/>Rejection-sampling fine-tuning RFT"]
    E --> F["TTS-Friendly Native Output<br/>Low latency directly to any TTS"]

Key Designs

1. Automated Feature Discovery and Frozen Response Scoring: Disentangling Interpretable Spoken Priors

Traditional preference alignment relies on monolithic, overparameterized neural reward models that behave as black boxes and easily overfit when training data is scarce. To avoid these issues, the method begins by discovering a set of \(F\) high-level linguistic features (\(F=40\) in practice) that capture contrasting properties between chosen (TTS-friendly) and rejected (TTS-unfriendly) utterances. By prompting an LLM with representative preference pairs from the target domain, the system surfaces explicit stylistic and structural descriptors, such as abbreviation_density, use_of_spelled_out_numbers, use_of_numeric_formatting, external_referral, and natural_conversational_tone. Each candidate response \(y\) for question \(x\) is then scored along each feature dimension using a frozen LLM scoring function \(\phi_f(x, y) \in [1, 5]\). Because the scoring functions require zero gradient updates or parameter tuning, this stage decouples feature representation from optimization and yields full attribution transparency.

2. Feature-aware Reward Model (FaRM): Pareto Tradeoff Learning in Parameter-Scarce Space

With response scores established across all feature dimensions, the Feature-aware Reward Model (FaRM) formulates the scalar reward as a linear combination of feature functions:

\[R_{\text{FaRM}}(x, y) = \sum_{f=1}^{F} \lambda_f \phi_f(x, y)\]

where each \(\lambda_f\) is an independently learned scalar weight. The entire reward model contains only 40 learnable parameters. The feature weights are trained on pairwise preference data using the standard Bradley-Terry objective, maximizing the likelihood of assigning higher rewards to chosen responses. This lightweight parameterization ensures rapid convergence and stability even when trained on as few as 10 examples. The learned weights explicitly quantify trade-offs between competing objectives: spoken clarity and conversational warmth receive positive weights (e.g., \(+0.33\) for spelled-out numbers), whereas compact written shorthands, raw URLs, and symbolic notation receive large negative penalties (e.g., \(-0.50\) for numeric formatting and \(-0.32\) for abbreviation density).

3. Rejection-Sampling Fine-Tuning (RFT): Single-Step Streaming with Zero Rewriting Latency

Once FaRM parameters are optimized, the framework updates the base generation model (evaluating Qwen3-4B and SmolLM3-3B) through iterative sampling and tuning. During each iteration, the generator samples multiple candidate responses for each domain prompt. FaRM evaluates and ranks these candidates according to its composite scalar reward, selecting the highest-scoring responses for policy optimization via rejection-sampling fine-tuning (RFT). Because the resulting model natively incorporates TTS constraints into its autoregressive decoding, it replaces written digits like $3.45 with spoken forms like three dollars and forty-five cents directly during the primary generation pass. This eliminates the downstream text normalization step entirely, cutting inference latency by half.

Loss & Training

  • FaRM Weight Optimization: Maximizes the Bradley-Terry log-likelihood over preference pairs \(\mathcal{D} = \{(x, y_w, y_l)\}\): $\(\mathcal{L}_{\text{FaRM}}(\{\lambda_f\}) = -\sum_{(x, y_w, y_l) \in \mathcal{D}} \log \sigma \left( \sum_{f=1}^{F} \lambda_f \left( \phi_f(x, y_w) - \phi_f(x, y_l) \right) \right)\)$ Optimized with L-BFGS or gradient descent at learning rate \(0.1\) for up to 500 iterations.
  • Policy Model Fine-Tuning (RFT): Standard autoregressive supervised loss on top-ranked responses \(\hat{y}\): $\(\mathcal{L}_{\text{RFT}}(\theta) = -\mathbb{E}_{(x, \hat{y})} \left[ \sum_{t=1}^{|\hat{y}|} \log P_\theta(\hat{y}_t \mid x, \hat{y}_{<t}) \right]\)$ Trained with AdamW, learning rate \(1.41 \times 10^{-5}\), batch size 16, across 5 outer sampling iterations with 5 fine-tuning epochs per iteration.

Key Experimental Results

Main Results

Evaluations are conducted on two newly curated benchmarks: CORA (262 customer-assistant dialogues for coffee ordering, 5-fold split with 100 train / 81 val / 81 test) and Recipe (300 procedural cooking instructions from RecipeNLG, 5-fold split with 100 train / 100 val / 100 test). Metrics include heuristic TTS-friendliness (1โ€“5 scale), synthesis-transcription round-trip character/word error rates (\(\text{TTS}\to\text{ASR}\) CER/WER via Kyutai Pocket TTS and Wav2Vec2-large), and task helpfulness (1โ€“5 scale via LLM-Judge). Table 1 presents generation results using the Qwen3-4B base model across the full 100-sample and 10-sample settings.

Approach Train Size CORA: Heur โ†‘ CORA: CER โ†“ CORA: WER โ†“ Recipe: Heur โ†‘ Recipe: CER โ†“ Recipe: WER โ†“
Oracle (Reference) - 4.838 0.051 0.260 4.799 0.044 0.228
Zeroshot (Base Qwen3-4B) 0 3.836 0.228 0.523 3.763 0.170 0.420
Prompting (TTS-rules) 0 3.999 0.206 0.478 4.324 0.123 0.349
DPO 100 4.010 0.141 0.377 4.079 0.117 0.297
FaST (Ours) 100 4.648 0.077 0.297 4.512 0.066 0.253

In runtime profiling, FaST single-pass generation requires 1.6s on CORA and 4.4s on Recipe, whereas cascading Zeroshot with PolyNorm text normalization requires 3.4s and 8.5s respectively, demonstrating that native alignment reduces latency by roughly 50%.

Ablation Study

Table 2 evaluates preference classification accuracy between the traditional neural reward model (RM) and FaRM across data regimes, alongside generation length controls.

Configuration / Reward Model Train Size CORA Acc โ†‘ Recipe Acc โ†‘ Gen. Heuristic โ†‘ Note
Traditional RM (4B params) 100 (full) 0.996 0.998 4.12 High pairwise accuracy but weaker generation guidance
FaRM (40 params) 100 (full) 0.984 0.914 4.73 Parameter-efficient, superior generation Pareto frontier
Traditional RM (4B params) 10 (low-data) 0.871 0.946 3.88 Degrades noticeably under severe sample scarcity
FaRM (40 params) 10 (low-data) 0.960 0.941 4.65 Robust classification and generation with 10 samples
FaST (default) 100 - - 4.73 Average response length: 434.6 characters
FaST (\(\lambda_{\text{conciseness}}=0\)) 100 - - 4.74 Length reduced to 335.6 characters without metric drop

Key Findings

  • FaST achieves exceptional sample efficiency in low-data regimes: With only 10 training examples, standard SFT and classical RM-based RL (GRPO/RFT) degrade significantly due to overfitting or weak reward signals. In contrast, FaRM's 40 scalar parameters maintain a 0.960 accuracy on CORA and keep generation quality stable.
  • Prompting alone cannot reliably enforce TTS friendliness: Supplying 25 detailed speech-writing rules in the system prompt modestly improves heuristic scores over Zeroshot, but base LLMs regularly ignore formatting constraints when dealing with complex units, prices, and timestamps.
  • Automatic heuristic metrics strongly mirror human perceptual listenability: In a double-blind MUSHRA listening test with 14 verified native raters on Prolific, FaST achieved a mean rating of 68.3, significantly outperforming DPO (55.9, Wilcoxon \(p=0.003\)) and Prompting (51.4, \(p=0.001\)). The heuristic score demonstrated a high pooled Spearman correlation of \(\rho = +0.84\) (\(p \ll 0.001\)) against human listenability ratings, confirming its utility as an efficient proxy.

Highlights & Insights

  • Internalizing speech normalization into generation: Instead of treating text normalization as an isolated engineering patch downstream, this work frames speakability as a native alignment objective, bypassing the latency and architectural overhead of cascaded pipelines.
  • Interpretable, parameter-scarce alignment: Replacing a 4-billion parameter neural reward model with 40 interpretable linguistic features and linear Bradley-Terry weights prevents overfitting in scarce data regimes while making alignment transparent.
  • Validated multi-tiered evaluation methodology: Establishing alignment between regex-based risk scores, round-trip ASR error rates, LLM helpfulness judges, and human MUSHRA listening tests provides a reliable protocol for speech-language research.

Limitations & Future Work

  • Verbosity bias: Because preference datasets naturally favor detailed, spelled-out spoken explanations over compact abbreviations, FaST learns negative weights for conciseness and brevity (\(-0.12\) on CORA, \(-0.25\) on Recipe). Although manually zeroing \(\lambda_{\text{conciseness}}\) effectively mitigates this issue, automated length-regularization requires further study.
  • Domain and language coverage: Empirical verification is restricted to English conversational coffee ordering and cooking recipes; expanding to technical, legal, and multilingual code-switching environments remains unexplored.
  • Cascade vs. end-to-end architectures: The approach is designed for decoupled text-to-speech pipelines (\(\text{LLM} \to \text{TTS}\)). Adapting feature-aware preference signals to latent acoustic tokens in unified multimodal speech LLMs is an open challenge.
  • vs PolyNorm (Wong et al., 2025): PolyNorm requires an additional few-shot LLM pass at inference time to rewrite text, doubling latency and inference compute. FaST generates TTS-friendly text natively in a single pass, cutting latency by 50%.
  • vs Speechworthy Tuning (Cho et al., 2024): Cho et al. explore broad speech suitability but conflate helpfulness with speechworthiness in human annotations; this work explicitly disentangles task helpfulness from speech-readiness using modular feature scoring and dedicated LLM judges.

Rating

  • Novelty: โญโญโญโญโ˜† Elegant formulation of TTS-friendliness as a multi-objective preference alignment problem with interpretable feature rewards.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation uniting regex heuristics, round-trip ASR, LLM helpfulness judges, and human MUSHRA studies.
  • Writing Quality: โญโญโญโญโญ Well-structured narrative, rigorous notation, and transparent analysis of feature weights.
  • Value: โญโญโญโญโญ Practical and cost-effective approach for real-time voice assistants and conversational agent pipelines.