Phrase-Localized Language-Contrastive Guidance: Training-Free Localized Accent Control for Code-Switching Text-to-Speech¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/saga1214/PhraseLocalizedLCG
Area: LLM Reasoning / Speech Synthesis / Cross-Lingual TTS
Keywords: Code-switching TTS, Cross-lingual accent leakage, Training-free inference guidance, Self-attention probing, Language-contrastive guidance
TL;DR¶
Addressing the severe cross-lingual accent leakage in zero-shot multilingual TTS where foreign embedded phrases are flattened into the matrix carrier accent during code-switching, this work introduces Phrase-Localized Language-Contrastive Guidance (LCG). LCG extracts acoustic phrase masks via internal self-attention probing and decouples contrastive language steering from global text guidance, delivering authentic native-accented pronunciations without fine-tuning.
Background & Motivation¶
The emergence of zero-shot multilingual text-to-speech (TTS) foundation models, particularly discrete diffusion language models (DLMs), has enabled high-fidelity cross-lingual voice cloning using short reference prompts without task-specific fine-tuning. However, when applied to intra-utterance code-switching (CS)—where a foreign embedded phrase is inserted into a primary matrix carrier language within a single utterance—these systems suffer from severe cross-lingual accent leakage. The foreign phrase is almost invariably pronounced with the phonetic rules, articulatory habits, and prosodic accent of the primary matrix language, leading to unnatural, distorted foreign pronunciations that alienate native listeners.
This localized pronunciation bottleneck stems from two deeply interconnected generative biases in current architectures. First, zero-shot voice cloning inherently entangles speaker identity with the reference prompt's native language habits, making it exceptionally difficult to disentangle vocal timbre from host language phonetics on a localized temporal segment. Second, Classifier-Free Guidance (CFG), universally adopted to enhance transcription fidelity and acoustic clarity, applies a uniform scalar multiplier across the entire temporal sequence. Standard global CFG lacks the spatial and temporal resolution required to modulate localized phonetic shifts; it aggressively amplifies the matrix reference voice bias globally and forcefully flattens local linguistic transitions into the dominant carrier accent.
Existing code-switching TTS frameworks attempt to resolve this by fine-tuning model weights via bilingual phonetic posteriorgrams, cross-lingual embeddings, multi-stage synthetic data training, or specialized multi-task diffusion modules. However, these methods incur prohibitive training costs, fail to scale, and struggle significantly when handling dense multi-word phrasal insertions. This paper approaches the challenge from a purely inference-time perspective. Core idea: adaptively probe internal self-attention layers of the discrete diffusion backbone to locate acoustic frame boundaries of the foreign phrase without external aligners, refine the candidate span via dual-tag mask union and symmetric margin dilation, and decouple an independent language-contrastive steering scale from global text guidance to inject authentic native pronunciations while relying on the bidirectional diffusion denoising buffer to absorb boundary errors.
Method¶
Overall Architecture¶
The proposed Phrase-Localized Language-Contrastive Guidance (LCG) framework is built upon pretrained discrete diffusion language models (DLMs), such as OmniVoice. The entire intervention operates during inference without modifying any model weights. As illustrated in the sequence pipeline, the generation process proceeds through three consecutive stages: (1) dynamic frame-level mask extraction via decoder internal self-attention probing, (2) asymmetric boundary expansion and mask refinement combining dual-tag union with symmetric margin dilation, and (3) localized scale adjustment via independent language-contrastive steering gating.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Code-Switching Sequence<br/>[lang_tag] [transcript] [ref_audio] [target_audio]"] --> B["Parallel Forward Logits Computation<br/>Matrix csrc / Foreign phrase cphr / Unconditional u"]
B --> C["Internal Attention Probing for Phrase Localization<br/>Monitor decoder self-attention at L8 and L12 via argmax"]
C --> D["Mask Refinement via Dual-Tag Union & Margin Dilation<br/>Combine msrc ∨ mphr and dilate with symmetric radius r_k"]
D --> E["Decoupled Contrastive Guidance with Independent Scaling<br/>Isolate λ(cphr - csrc) from global text guidance scale γ"]
E --> F["Gated Final Denoising Logits Synthesis<br/>l_LCG = csrc + γ(csrc - u) + m * λ(cphr - csrc)"]
F --> G["Discrete Diffusion Bidirectional Denoising Iterations<br/>Buffer absorbs transient boundary false positives to output speech"]
Key Designs¶
1. Internal Attention Probing for Phrase Localization: Training-Free Acoustic Frame Alignment Conventional speech processing relies on heavy external forced-alignment models (e.g., Montreal Forced Aligner or offline ASR models) to locate phonetic time boundaries, introducing auxiliary latency and failing to operate seamlessly on intermediate non-autoregressive decoding states. This work observes that the intermediate decoder layers of discrete diffusion language models implicitly preserve high-fidelity acoustic-to-text alignments. During each forward unmasking step, the framework monitors the self-attention tensor across decoder layers and identifies the text token receiving maximum attention weight for each generated acoustic frame \(a\) via \(\operatorname{argmax}_t A[a, t]\). If this token falls within the character span of the embedded foreign phrase, the frame is assigned \(m[a] = 1\); otherwise \(m[a] = 0\). Empirical probing on English (LJSpeech) and Korean (KSS) pilot corpora reveals that alignment tracking spikes sharply in mid-decoder layers, specifically layers 8 and 12. A multi-layer ensemble over \(\{L8, L12\}\) combined with head-wise max-pooling maximizes the worst-case boundary recall across structurally diverse language families, bypassing external aligners completely.
2. Mask Refinement via Dual-Tag Union & Margin Dilation: Asymmetric Alignment Error Tolerance Localized acoustic steering is governed by a pronounced precision-recall asymmetry: false negatives (missed frames within the embedded phrase) are catastrophic because unguided frames immediately default to the matrix carrier accent, triggering irreversible pronunciation collapse; conversely, false positives (erroneously guided frames spilling into the matrix carrier) are benign because the dominant textual and acoustic context of the surrounding carrier naturally overrides accidental steering, and transient boundary shifts are safely smoothed out during iterative bidirectional diffusion denoising. To capitalize on this asymmetry, the framework applies two zero-overhead refinement operations: first, it computes a dual-tag logical union \(m_{XU} = m_{src} \lor m_{phr}\), merging alignments extracted under the matrix prefix and the foreign phrase prefix to capture subtle boundary shifts across tag conditions; second, it applies a symmetric graded margin dilation with cumulative triangular radius \(r_k = \frac{k(k+1)}{2}\) via a max-filter: $\(m[a] = \max_{|\delta| \le r_k} m_{XU}[a + \delta]\)$ At the optimal configuration \(k = 4\) (covering an effective window of \(\pm 10\) frames, denoted as \(M_{4+XU}\)), boundary recall is maximized. The diffusion model's bidirectional denoising buffer effectively insulates the carrier against over-dilation artifacts, ensuring robust foreign phonetics without compromising carrier fluency.
3. Decoupled Contrastive Guidance with Independent Scaling: Breaking Standard CFG Coupling An intuitive baseline, termed Swap, routes the foreign-conditioned logit \(c_{phr}\) inside the mask (\(m[a]=1\)) and the matrix-conditioned logit \(c_{src}\) outside (\(m[a]=0\)), leading to \(\tilde{c} = (1 - m) c_{src} + m c_{phr}\). Integrating this into standard CFG yields: $\(\tilde{l}_{swap} = [c_{src} + \gamma(c_{src} - u)] + m (1 + \gamma)(c_{phr} - c_{src})\)$ Algebraic decomposition proves that Swap is mathematically equivalent to maintaining global carrier CFG while injecting a localized corrective vector pointing in the contrastive direction \((c_{phr} - c_{src})\). However, Swap rigidly couples the contrastive steering magnitude to the global text guidance scale via the fixed coefficient \((1 + \gamma)\), preventing sufficient localized accent enforcement. LCG resolves this limitation by substituting the implicit scale with an independent control parameter \(\lambda\): $\(\tilde{l}_{LCG} = [c_{src} + \gamma(c_{src} - u)] + m \lambda (c_{phr} - c_{src})\)$ By decoupling \(\lambda\) from global text scale \(\gamma = 2.0\), LCG allows independent amplification of the localized contrastive steering vector (empirically optimized at \(\lambda = 7\)). This forces the acoustic features within the gated span toward the authentic phonetics of the foreign language without destabilizing global carrier fluency or speaker identity.
Benchmark & Evaluation Protocol¶
To benchmark code-switching TTS, the authors construct a balanced, 1,200-utterance synthetic evaluation corpus generated via GPT-5.5 with reasoning effort across 5 languages (English, German, French, Japanese, Korean) perfectly balanced over 12 bi-directional language pairs (100 utterances each). Each utterance contains 3–5 dense, multi-word phrasal insertions (mean \(\sim 3.6\) phrases, \(\sim 6\) words, \(\sim 35\) characters per utterance) spanning 18 technical/literary domains and 6 stylistic registers. Monolingual reference voices from standard datasets are fixed per direction to eliminate identity confounding. Evaluation combines objective speech metrics—Mixed Error Rate (MER, evaluated as WER for Latin scripts and CER for Japanese/Korean), Language Accuracy (\(LA_e\)), Language Identification Confidence (\(LID_e\)), Speaker Similarity via WavLM (global SIM and intra-utterance consistency \(SIM_{m \leftrightarrow e}\)), and UTMOS naturalness—with rigorous crowdsourced human listening tests on Amazon Mechanical Turk covering 5-point Global Quality MOS and forced-choice Phrase Nativeness AB Preference.
Key Experimental Results¶
Main Results¶
Across the 1,200-utterance 12-direction evaluation corpus, Phrase-Localized LCG (\(M_{4+XU}, \lambda = 7\)) achieves dramatic improvements in embedded-phrase nativeness while preserving global acoustic naturalness and speaker identity compared to the unguided baseline (Vanilla CFG, \(\lambda = 0\)):
| Metric / Evaluation Dimension | Baseline (base) | LCG (Ours, \(\lambda=7\)) | Relative / Absolute Gain \(\Delta\) |
|---|---|---|---|
| Mixed Error Rate (MER) ↓ | 0.564 | 0.445 | -0.119 (-21.1%) |
| Language Accuracy (\(LA_e\)) ↑ | 0.233 | 0.518 | +0.285 (+122.3%) |
| Language Identification (\(LID_e\)) ↑ | 0.247 | 0.588 | +0.341 (+138.1%) |
| Predicted Naturalness (UTMOS) ↑ | 3.411 | 3.285 | -0.126 |
| Speaker Similarity (SIM) ↑ | 0.973 | 0.969 | -0.004 |
| Intra-Utterance Consistency (\(SIM_{m \leftrightarrow e}\)) ↑ | 0.961 | 0.959 | -0.002 |
| Human Quality MOS (5-point) ↑ | 4.007 ± 0.110 | 3.931 ± 0.092 | Statistically indistinguishable (overlapping 95% CIs) |
| Human Phrase Nativeness AB Preference ↑ | 19.5% (95 wins) | 59.8% (292 wins) | 75.5% win rate excluding ties (\(p < 0.001\)) |
Ablation Study: Mask Source & Boundary Refinement¶
Fixing the contrastive guidance scale at \(\lambda = 7\), the table below compares the baseline, the precision-heavy offline forced-alignment oracle (Qwen3-ForcedAligner-0.6B), and the internal attention probe across various dilation widths (\(k \in \{0, 2, 4\}\)) and dual-tag union (\(XU\)):
| Configuration | Mask Source / Refinement Description | MER ↓ | \(LA_e\) ↑ | \(LID_e\) ↑ | UTMOS ↑ |
|---|---|---|---|---|---|
| no_CS (Baseline) | Unguided global CFG (\(\lambda = 0\)) | 0.564 | 0.233 | 0.247 | 3.411 |
| gt (Oracle Aligner) | Offline Qwen3-ForcedAligner-0.6B | 0.472 | 0.459 | 0.526 | 3.333 |
| \(k=0\) (base probe) | Raw argmax attention mask (\(r_0=0\)) | 0.455 | 0.499 | 0.568 | 3.325 |
| \(k=0 + XU\) | Raw argmax mask + dual-tag union \(m_{XU}\) | 0.452 | 0.493 | 0.570 | 3.329 |
| \(k=2\) | Single-tag probe + dilation (\(r_2=3\) frames) | 0.455 | 0.514 | 0.587 | 3.305 |
| \(k=2 + XU\) | Dual-tag union + dilation (\(r_2=3\) frames) | 0.447 | 0.517 | 0.582 | 3.304 |
| \(k=4\) | Single-tag probe + dilation (\(r_4=10\) frames) | 0.450 | 0.509 | 0.584 | 3.288 |
| \(k=4 + XU\) (Ours) | Dual-tag union + dilation (\(r_4=10\) frames) | 0.445 | 0.518 | 0.588 | 3.285 |
Key Findings¶
- The Mask Source Paradox: The internal attention-probed mask with refinement (\(M_{4+XU}\)) decisively outperforms the precision-heavy offline forced-alignment oracle (MER 0.445 vs. 0.472, \(LA_e\) 0.518 vs. 0.459). The forced aligner prioritizes boundary precision at the expense of transitional recall, truncating boundary frames at code-switching junctions and causing localized pronunciation collapse. By contrast, expanded attention probing ensures high boundary recall, while over-expanded false-positive frames are naturally absorbed by the diffusion model's bidirectional denoising buffer.
- Guidance Intensity Pareto Knee at \(\lambda = 7\): Parameter sweeping over \(\lambda \in \{0, 3, 5, 7, 9, 11, 13\}\) reveals that foreign accent metrics (\(LA_e, LID_e\)) exhibit a logarithmic saturation curve, rising rapidly up to \(\lambda = 7\) before plateauing. Meanwhile, speech naturalness (UTMOS) degrades linearly and drops sharply beyond \(\lambda = 7\). Setting \(\lambda = 7\) maximizes the Pareto frontier between localized foreign accent enforcement and global acoustic naturalness.
- Generalization to Human-Authored Conversations: On real-world Korean-English conversational transcripts from the AI-Hub dataset, LCG maintains the exact performance hierarchy (Baseline < Swap < LCG) across both dense multi-word insertions and single-word code-switching regimes. Furthermore, comparing against MOSS-TTS-v1.5 (an 8B model natively supporting code-switching), MOSS-TTS achieves only \(LA_e = 0.280\), demonstrating that scaling model size under global utterance-level language conditioning cannot resolve accent leakage without localized steering.
Highlights & Insights¶
- Repurposing Self-Attention Probing as an Active Inference Controller: Rather than treating attention weights as a passive interpretability diagnostic, LCG exploits decoder self-attention alignments on-the-fly to construct zero-overhead acoustic guidance masks, eliminating dependencies on external aligners.
- Exploiting Asymmetric Error Tolerance in Generative Diffusion: The realization that boundary recall loss is fatal while boundary precision loss is benign under bidirectional diffusion denoising provides a profound theoretical justification for expanded margin dilation in localized generative control.
- Elegant Algebraic Decoupling of CFG Dynamics: Decomposing intuitive tag swapping into global text guidance plus a localized contrastive direction exposes the hidden scalar bottleneck in standard CFG and unlocks flexible, continuous accent modulation.
Limitations & Future Work¶
- Architectural Coupling with Discrete Diffusion Backbones: The framework relies fundamentally on the non-autoregressive bidirectional denoising buffer of discrete diffusion models to absorb mask boundary spillovers. Applying localized contrastive steering to autoregressive (AR) speech language models risks compounding autoregressive error cascades.
- Requirement of Character-Level Phrase Spans: The framework assumes prior knowledge of the foreign phrase's character span in the text transcript. While easily derived from Unicode script differences in Latin-JK pairs, intra-script mixing (e.g., English-German) requires an upstream text language identification step.
- Attention Computation Backend Constraint: Because LCG accesses attention weight matrices directly, it requires falling back to eager attention execution during forward passes, bypassing fused kernels like FlashAttention or SDPA and increasing total synthesis time by \(2.26\times\) (though still running \(\sim 19\times\) faster than real time).
Related Work & Insights¶
- vs. X-Voice (Xu et al., 2026): X-Voice introduces decoupled CFG paths across generation timesteps with decay schedules and requires massive multilingual fine-tuning, yet still struggles on intra-sentential transitions; LCG operates strictly at inference time with zero training cost, offering fine-grained frame-level spatial guidance.
- vs. Selective CFG (Zheng & Maleki, 2026): Selective CFG applies guidance masks exclusively at the discrete text token level; LCG bridges the cross-modal gap by projecting dynamic alignments onto continuous acoustic frame representations.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering training-free localized accent control combining attention probing and decoupled contrastive CFG]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Balanced 1,200-utterance 12-direction benchmark, ablation studies, real human transcripts, and crowdsourced AMT human validations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical formulation, clear narrative structure, and compelling empirical analyses]
- Value: ⭐⭐⭐⭐⭐ [Provides a practical, plug-and-play inference solution for cross-lingual accent leakage in foundation speech models]