Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Pooookeman/LoopTTS
Area: LLM Reasoning
Keywords: Text-to-Speech, Closed-Loop Refinement, AudioLLM, Prosodic Correction, Weak Supervision
TL;DR¶
LOOPTTS establishes a closed-loop TTS framework featuring a three-stage Filter–Judge–Refiner pipeline, where an AudioLLM diagnoses subtle word-level prosodic defects to produce structured instructions, and a dedicated Refiner trained with position-aware objectives performs guided expressive re-synthesis to resolve sporadic failures.
Background & Motivation¶
The convergence of discrete neural audio codecs and large language model architectures has established a powerful paradigm for text-to-speech (TTS) synthesis, achieving near-human naturalness and flexible zero-shot cloning across varied domains. Contemporaneous developments in controllable speech synthesis have unlocked multi-attribute conditioning, encompassing emotion control, expressive natural-language style prompts, and coarse instruction-guided voice modulation. Despite these impressive strides, achieving consistently high acoustic and prosodic fidelity at production scale remains an open challenge. The overwhelming majority of deployed TTS systems adhere strictly to an open-loop, single-pass generation design: upon generating an initial speech sample, the pipeline either accepts the raw waveform or, if an anomaly is detected, discards the entire generation and resynthesizes from scratch using an arbitrary random seed.
Due to the inherent stochasticity of autoregressive token sampling and flow-matching decoding, even state-of-the-art TTS models sporadically produce local, highly jarring prosodic defects. These subtle flaws include misplaced lexical stress, contextually jarring pauses, and unnaturally flattened pitch contours. While catastrophic degradations are easily flagged by utterance-level metrics such as Word Error Rate (WER), UTMOS, or cosine speaker similarity, fine-grained prosodic errors often slip through unnoticed because they preserve verbal intelligibility while severely degrading perceived naturalness—such as stressing an unstressed grammatical particle. Existing attempts to incorporate AudioLLMs into TTS post-training (e.g., LLM-as-a-Judge and RLAIF) collapse rich acoustic evaluations into scalar preference scores, completely discarding the precise diagnostic details required for targeted correction (e.g., identifying the exact token index where emphasis should be placed). Meanwhile, conventional speech editing frameworks focus predominantly on waveform-level text replacement or insertion, offering no mechanism to interpret nuanced, sentence-wide prosodic instructions.
To bridge this fundamental divide between diagnostic assessment and targeted execution, this work investigates how an AudioLLM can serve as an active diagnostic judge that emits structured, actionable refinement directives to guide a specialized re-synthesis model. Core idea: Formulate a closed-loop Filter–Judge–Refiner architecture (LOOPTTS) where an AudioLLM diagnoses salient word-level prosodic defects (stress and pause) to produce structured JSON refine instructions, and train an instruction-following Refiner using contrastive weak supervision and position-aware objectives to perform guided expressive re-synthesis.
Method¶
Overall Architecture¶
The LOOPTTS framework operates as an offline quality-assurance pipeline structured around three sequential stages (Filter, Judge, and Refiner), complemented by a contrastive weak supervision paradigm for data construction. Given a target text and style request, a base TTS model first generates an initial utterance. Stage 1 applies coarse-grained filtering to eliminate degenerate audio. Stage 2 employs an AudioLLM Judge to diagnose prosody and generate structured instructions for low-scoring samples. Finally, Stage 3 leverages the Refiner to execute expressive re-synthesis conditioned on the initial audio, the refine instruction, and the text.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Target Text & Style Request"] --> B["Stage 1: Coarse-Grained Filtering<br/>WER & UTMOS Gate with Retry"]
B --> C["Stage 2: AudioLLM Prosodic Diagnosis<br/>Gemini-3-pro Scoring & Structured JSON"]
C -->|Score >= 5| D["Accepted as Final Output"]
C -->|Score < 5 Flagged| E["Contrastive Instruction Data Construction<br/>REFINER-DB with 42K Contrastive Tuples"]
E --> F["Position-Weighted Cross-Entropy Loss<br/>Timestamp Alignment & Gradient Amplification"]
F --> G["Position-Aware Structural Alignment Loss<br/>Within-Span RSM Relational Consistency"]
G --> H["Stage 3: Guided Expressive Re-Synthesis<br/>Refiner Generates Corrected Utterance"]
H -->|Optional Iteration Round 2| C
Key Designs¶
1. Cascaded Quality Filtering and AudioLLM Diagnosis: Decoupling Severe Failures from Fine-Grained Prosodic Defects
Directly deploying a heavyweight multimodal AudioLLM on all synthesized samples incurs prohibitive inference latency and cost. LOOPTTS resolves this bottleneck by decoupling quality assurance into Stage 1 (Filter) and Stage 2 (Judge). Stage 1 uses Whisper-large-v3-turbo for Word Error Rate (WER) computation and a lightweight UTMOS estimator: utterances pass only when \(\text{WER} \le 3.0\%\) and \(\text{UTMOS} > 3.0\). Any sample failing these criteria triggers up to two full re-generation retries with different random seeds, filtering out catastrophic word omissions or severe audio garbling at negligible computational overhead. Qualified audio then proceeds to Stage 2, where an AudioLLM Judge evaluates prosodic naturalness and emotional expression on a 1–10 scale. Utterances scoring at least \(\tau_{\text{judge}} = 5\) are immediately accepted; lower-scoring outputs are flagged and assigned a structured JSON instruction detailing global attributes (emotion, speed, pitch from a strict enum) and local edits (salient word-level stress and pause locations), along with a concise natural-language sentence \(I\). This cascade confines expensive LLM processing strictly to borderline cases requiring prosodic remediation.
2. Contrastive Instruction Data Construction: Overcoming the Word-Level Prosodic Annotation Bottleneck
Supervising the Refiner to faithfully follow local prosodic instructions demands thousands of paired examples with word-level stress and pause annotations, which are virtually impossible to acquire via exhaustive manual labeling due to inter-annotator disagreement and high costs. The authors introduce a contrastive weak-supervision pipeline to build REFINER-DB (~42K tuples). Sourcing expressive human recordings \(y_{\text{tgt}}\) from six public corpora (ESD, RAVDESS, SAVEE, MESS, EmoVoice-DB, LibriTTS), they synthesize a paired neutral counterpart \(y_{\text{neu}}\) using CosyVoice2 zero-shot speaker cloning with the same text and speaker prompt (filtering pairs to guarantee \(\text{WER} = 0\) and \(\text{UTMOS} > 3\)). An AudioLLM (Gemini-3-pro) then listens to both the flat synthetic audio and the expressive human recording in tandem, identifying the top 3–5 most prominent acoustic discrepancies in stress and pause placement. Merging global metadata with local contrastive annotations yields rich training tuples \(\mathcal{D} = \{(y_{\text{neu}}, I, y_{\text{tgt}}, T)\}\) at an estimated construction cost of under $910.
3. Position-Weighted Cross-Entropy Loss: Countering Gradient Dilution on Sparse Prosodic Spans
In a standard speech sequence of 200+ neural codec tokens, the tokens corresponding to instruction-specified stress or pause positions typically occupy only 10–30 frames. When optimizing standard cross-entropy, these sparse, critical signals are heavily diluted by the overwhelming majority of unmodified background tokens, leading the model to ignore fine-grained directives. Using whisper-timestamped word alignment, the framework identifies exact token spans corresponding to targeted words and pauses, assigning each token a prosodic role label \(m_t \in \{0, 1, 2\}\) (0: background, 1: stress span, 2: pause span). The position-weighted loss dynamically scales the gradient for each frame:
By setting \(\alpha = 0.2\) and \(\beta = 0.3\), the loss mildly amplifies optimization signals over annotated local edits without destabilizing global acoustic modeling or shifting overall speech pacing.
4. Position-Aware Structural Alignment Loss: Preserving Span-Level Relational Consistency
While position-weighted cross-entropy penalizes isolated token mispredictions, acoustic stress realization is fundamentally a multi-token phenomenon spanning 5–20 contiguous frames characterized by coordinated pitch elevation, duration expansion, and energy shifts. Independent point-wise token supervision can predict correct token categories in isolation while disrupting the internal acoustic coherence of the stressed segment. To enforce structural continuity, the authors introduce a relational alignment objective based on Representational Similarity Matrices (RSMs). For each maximal contiguous stress span \(S \in \mathcal{P}\), acoustic-code hidden representations \(H_S \in \mathbb{R}^{|S| \times D}\) are row-wise \(\ell_2\)-normalized to construct a pairwise cosine similarity matrix \(R_S = \bar{H}_S \bar{H}_S^\top \in \mathbb{R}^{|S| \times |S|}\). The structural loss minimizes the squared Frobenius distance between predicted representations and target representations, normalized by span area:
This geometric constraint encourages the model hidden states to match the relational topology of expressive human stress realization, preventing abrupt timbre artifacts.
Loss & Training¶
The Refiner is built upon EmoVoice-1.5B (using Qwen2.5-1.5B as the language model backbone), predicting 50 Hz CosyVoice grouped semantic codes that are decoded to waveforms via flow matching and HiFi-GAN. Fine-tuning is conducted on all LLM parameters while freezing the audio encoder. The input sequence is formatted as \([I; y_{\text{init}}; T]\), where the instruction \(I\) is placed first to condition all subsequent tokens under causal self-attention, and \(y_{\text{init}}\) serves as the acoustic reference. The combined training objective is:
with \(\lambda = 0.1\). The model is trained with AdamW (\(\text{lr} = 5 \times 10^{-6}\), 1,000 warmup steps, 300K total steps, batch size 8). Inference uses deterministic greedy decoding with a maximum generation limit of 256 new tokens.
Key Experimental Results¶
Main Results¶
The full LOOPTTS pipeline is evaluated on 200 Stage 2 flagged utterances (100 Neutral, 100 Emotional) drawn from CosyVoice2 generations. Ten professional evaluators conducted blind shuffled listening tests. Systems are evaluated on Word Error Rate (WER), WavLM speaker similarity against neutral reference (SIM), and subjective Mean Opinion Score (MOS, 1–5 scale).
| Condition & System | Neutral WER↓ | Neutral SIM↑ | Neutral MOS↑ | Emotional WER↓ | Emotional SIM↑ | Emotional MOS↑ |
|---|---|---|---|---|---|---|
| Baseline (Before refinement) | 2.04 | 0.73 | 3.21±.15 | 2.11 | 0.69 | 3.01±.16 |
| Open-loop Regen: CosyVoice2 | 1.95 | 0.71 | 3.45±.14 | 2.03 | 0.70 | 3.15±.15 |
| Open-loop Regen: EmoVoice | 1.83 | 0.68 | 3.90±.13 | 1.72 | 0.65 | 3.85±.12 |
| LOOPTTS (Global-only instructions) | 1.87 | 0.68 | 3.96±.10 | 1.82 | 0.66 | 3.72±.16 |
| LOOPTTS (1-round, Gemini-3-pro) | 1.86 | 0.68 | 4.17±.12 | 1.79 | 0.67 | 4.01±.14 |
| LOOPTTS-Q (1-round, Qwen3-Omni) | 1.90 | 0.67 | 4.09±.16 | 1.84 | 0.66 | 3.98±.15 |
| LOOPTTS (2-round refinement) | 1.85 | 0.66 | 4.16±.13 | 1.80 | 0.64 | 4.17±.13 |
| Human-instruction (Soft Upper Bound) | 1.79 | 0.67 | 4.20±.14 | 1.78 | 0.67 | 4.26±.12 |
In paired A/B preference evaluations over 30 samples, LOOPTTS achieved an 85.7% win rate against raw unrefined audio, a 77.0% win rate against CosyVoice2 full re-generation, and a 72.3% win rate against EmoVoice re-generation (with 19/30 samples reaching at least 7/10 listener consensus).
Ablation Study¶
Ablations on training loss components and conditioning inputs isolate the contribution of structural alignment, position reweighting, and acoustic anchoring:
| Configuration | WER↓ | SIM↑ | MOS↑ | Avg. MOS-I↑ | Description |
|---|---|---|---|---|---|
| Full Refiner | 2.90 | 0.68 | 4.21±.14 | 4.21±.09 | Incorporates both \(\mathcal{L}_{\text{struct}}\) and position weighting |
| w/o \(\mathcal{L}_{\text{struct}}\) | 2.78 | 0.68 | 4.06±.15 | 4.01±.09 | Largest drop in MOS-I; degraded stress span coherence |
| w/o Position-Weighting | 2.92 | 0.68 | 4.17±.14 | 4.13±.08 | Moderate decrease across prosodic instruction following |
| w/o Both losses | 2.87 | 0.68 | 4.03±.14 | 3.75±.10 | Severe degradation in fine-grained instruction following |
| w/o \(y_{\text{init}}\) conditioning | 2.91 | 0.62 | 4.17±.12 | 3.94±.11 | SIM drops precipitously from 0.68 to 0.62 |
Key Findings¶
- Structural alignment drives prosodic adherence: Removing the RSM-based structural alignment loss \(\mathcal{L}_{\text{struct}}\) results in the largest drop in instruction following (Avg. MOS-I plunges from 4.21 to 4.01), confirming that preserving relational geometric structure across multi-frame stress spans is vital for natural emphasis realization.
- Acoustic conditioning prevents identity drift: Omitting the defective initial utterance \(y_{\text{init}}\) leads to a severe drop in speaker similarity (SIM drops from 0.68 to 0.62), demonstrating that conditioning on the initial audio anchors speaker identity and unmodified prosodic regions, preventing arbitrary deviation.
- Diminishing returns beyond two iterations: Scaled evaluations across multiple refinement rounds indicate that perceptual gains saturate by round 2 (MOS increases from 4.01 to 4.17 in emotional speech), while a third round yields no additional MOS improvement and exacerbates speaker embedding drift (SIM falls to 0.58), positioning 1–2 rounds as the optimal operating budget.
Highlights & Insights¶
- From Passive Reranking to Active Prescriptive Diagnosis: Unlike prior preference alignment pipelines that treat AudioLLMs merely as scalar evaluators, LOOPTTS turns the AudioLLM into an actionable diagnostician producing structured, executable tokens for downstream remediation.
- Scalable Contrastive Weak Supervision: By pairing synthetic neutral audio with expressive natural recordings, the authors circumvent the bottleneck of manual phonetics annotation, establishing an automated weak-supervision pipeline costing under $1,000 for 42K examples.
- Robustness Against Erroneous Over-Diagnosis: Counterfactual stress and pause evaluations reveal that the Refiner does not blindly execute unnatural directives (only a 10% erroneous execution rate on corrupted stress cues), demonstrating that pretraining and target acoustic priors prevent arbitrary distortion.
Limitations & Future Work¶
- Expressiveness–Similarity Trade-off: Because the Refiner performs guided expressive re-synthesis rather than localized waveform splicing, intensive prosodic restructuring and heightened emotional dynamics can shift global speaker embeddings, mildly lowering neutral speaker similarity metrics.
- Binary Prosodic Control: The current framework represents stress and pause directives as binary decisions, lacking continuous-scale modulation for nuanced stress intensity or millisecond-level duration control.
- Language and Corpus Scope: Empirical validations are currently confined to English read and emotional speech; evaluating the framework on tonal languages (e.g., Mandarin) and conversational overlapping speech represents a vital frontier.
Related Work & Insights¶
- vs Open-Loop Re-Generation (CosyVoice2 / EmoVoice): Retrying synthesis with a different random seed is untargeted and often re-introduces identical or new defects; LOOPTTS confines edits to diagnosed locations while preserving well-formed segments.
- vs Speech Editing Methods (VoiceCraft / SpeechX): Classical speech editing models operate on content-level replacement and inpainting, lacking the ability to parse high-level pragmatic and prosodic instructions like adding subtle emphasis to specific lexical items.
- vs Prompt-Only TTS Baselines (Qwen3-TTS / MOSS-TTS): Experiments show that state-of-the-art open-source TTS models cannot reliably execute fine-grained word-level stress or pause modifications through free-form text prompting alone; specialized fine-tuning with position-aware objectives remains necessary.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering closed-loop, diagnosis-driven corrective framework for neural TTS.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive blend of 3,000-utterance scaled objective evaluations, blind human listening tests, counterfactual audits, and multi-round analysis.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous mathematical definitions, and thorough appendix documentation.
- Value: ⭐⭐⭐⭐⭐ Provides an immediately deployable blueprint for offline quality assurance and automated defect correction in industrial speech pipelines.