Instance-Specific Test-Time Training for Speech Editing in the Wild

Conference: NeurIPS 2025 (Workshop on GenProCC) arXiv: 2506.13295 Code: To be confirmed Area: Speech Processing / Speech Editing Keywords: speech editing, test-time training, bandwidth discontinuity, phoneme prediction, in-the-wild adaptation

TL;DR

This paper proposes an instance-specific test-time training (TTT) method for in-the-wild speech editing. Prior to inference, the model is fine-tuned at the instance level using direct supervision from the acoustic features of unedited regions, and indirect supervision over edited regions via duration constraints and a phoneme prediction auxiliary loss. This approach effectively mitigates bandwidth discontinuity at editing boundaries, supports precise speaking-rate control through mask length adjustment, and surpasses existing systems on both objective and subjective metrics on an in-the-wild benchmark.

Background & Motivation

Background: Speech editing systems (FluentSpeech, VoiceBox, A3T, etc.) aim to naturally modify speech content while preserving acoustic consistency and speaker identity. The dominant paradigm is mask-and-predict — masking the region to be edited and using a model to predict new acoustic features.

Limitations of Prior Work: (a) Acoustic environment diversity: training data typically comes from clean studio recordings, whereas real-world audio contains diverse background noise, reverberation, and device characteristics, causing domain-shift failures; (b) Bandwidth discontinuity: edited regions (model-generated, band-limited) and unedited regions (original audio, full-bandwidth) exhibit abrupt spectral differences, producing perceptible discontinuity artifacts at boundaries; (c) Imprecise speaking-rate control: the speaking rate of edited regions is determined internally by the model and is difficult to match to a target duration precisely.

Key Challenge: Speech editing must simultaneously satisfy acoustic consistency (matching the original audio) and content correctness (generating the correct speech content), yet under unseen acoustic conditions it is difficult to guarantee both — models tend to generate edited regions that are "clean but acoustically mismatched" with the original.

Goal: Enable a speech editing system to adapt to the acoustic environment of each individual test sample without retraining the main model.

Key Insight: Test-time training (TTT) — fine-tuning the model using information contained within each test sample itself (the acoustic features of unedited regions), thereby adapting it to the sample's specific acoustic conditions. This is the first attempt to introduce TTT into the speech editing domain.

Core Idea: Direct supervision from unedited regions + indirect constraints on edited regions (duration + phonemes) = instance-level acoustic adaptation.

Method

Overall Architecture

A small number of gradient update steps are performed on each test sample before inference: (1) in unedited regions, the model reconstructs the original acoustic features (mel-spectrogram, etc.) with a loss that directly optimizes acoustic matching; (2) in edited regions, two auxiliary tasks provide constraints — a duration loss ensures the output matches the target duration determined by the mask length, and a phoneme prediction loss back-predicts the phoneme sequence from the acoustic features to ensure content correctness; (3) after adaptation, speech editing proceeds normally.

Key Designs

1. Direct Supervision from Unedited Regions:

   • Function: Instance-level fine-tuning of the editing model using acoustic features from the original audio.
   • Mechanism: For time frames in unedited regions, an L1/L2 loss is computed between the model's reconstructed output and the original mel-spectrogram. This forces the model to learn the acoustic environment characteristics of the current sample (noise level, reverberation, recording device properties, etc.).
   • Design Motivation: Unedited regions provide a free "self-supervised signal" — the model is expected to faithfully reproduce these regions. By adapting to their acoustic properties, the model can maintain a consistent acoustic style when generating the edited region.

2. Indirect Supervision for Edited Regions:

   • Function: Provides soft constraints in edited regions where no ground-truth target exists.
   • Mechanism: (a) Duration constraint — the predicted duration of the edited region should match the mask length, simultaneously enabling speaking-rate control; (b) Phoneme prediction — the phoneme sequence back-predicted from the generated acoustic features should be consistent with the target text, ensuring content correctness.
   • Design Motivation: Direct supervision cannot be applied to edited regions (no ground truth), but quality can be ensured through downstream verification. The duration constraint also yields an additional benefit: precise speaking-rate control over the edited region is achieved simply by adjusting the mask length.

3. Bandwidth Discontinuity Mitigation:

   • Function: Resolves abrupt spectral transitions at editing boundaries.
   • Mechanism: Through instance-level adaptation, the model learns to generate edited regions whose spectral characteristics — including high-frequency components — match those of the original audio. Direct supervision on unedited regions implicitly transfers bandwidth information of the current sample to the model.
   • Design Motivation: Bandwidth discontinuity is one of the most perceptible artifacts in speech editing, and conventional post-processing approaches (e.g., transition smoothing) offer limited improvement.

Loss & Training

• Total test-time loss: \(\mathcal{L} = \alpha \mathcal{L}_{direct} + \beta \mathcal{L}_{duration} + \gamma \mathcal{L}_{phoneme}\)
• A small number of gradient steps (approximately 20–50) are performed independently for each test sample.
• Only a subset of decoder/acoustic model parameters are fine-tuned; the encoder is kept frozen.

Key Experimental Results

Main Results

Evaluated on an in-the-wild benchmark dataset containing various background noise conditions and recording environments:

Method	MOS	PESQ	WER	Boundary Smoothness
FluentSpeech	Baseline	Baseline	Baseline	Baseline
VoiceBox	Moderate	Moderate	Moderate	Moderate
Ours (TTT)	Best	Best	Best	Best

Ablation Study

Configuration	Performance	Note
Full model	Best	All three losses combined
w/o direct supervision	Significant drop	Acoustic adaptation fails
w/o duration constraint	Slight drop	Speaking-rate mismatch
w/o phoneme prediction	Drop	Content errors may occur

Key Findings

• Direct supervision is the most critical component: the reconstruction loss on unedited regions provides the strongest adaptation signal.
• There is an optimal number of TTT steps: too few yields insufficient adaptation; too many causes overfitting to the current sample.
• Bandwidth discontinuity is substantially improved: spectrogram visualizations show smoother transitions at editing boundaries.
• Speaking-rate control is precise: rate adjustment is achieved by combining mask-length modification with the duration constraint.

Highlights & Insights

• The insight that "the test sample itself is the best adaptation data" is profound — the unedited portion of the same audio clip naturally provides a complete description of the current acoustic environment.
• This is the first application of test-time training to speech editing — the approach is generalizable to other audio generation tasks (e.g., speaker adaptation for TTS, music editing).
• Controlling speaking rate via mask length is more elegant than incorporating a separate duration prediction module.

Limitations & Future Work

• As a workshop paper, the scale is limited and experimental validation is not comprehensive.
• TTT increases inference time (tens of gradient update steps per sample), making it unsuitable for real-time scenarios.
• Validation is limited to speech editing; applicability to TTS and voice conversion remains unexplored.
• Performance under extreme acoustic conditions (e.g., very low SNR) has not been tested.
• The strategy of fine-tuning only a subset of parameters lacks theoretical justification.

Related Work & Insights

• vs. FluentSpeech/VoiceBox: These methods suffer significant performance degradation in cross-domain scenarios; this paper achieves zero-shot domain adaptation via TTT.
• vs. Speaker Adaptation for TTS: Speaker adaptation in TTS typically requires several minutes of reference speech, whereas this method adapts from the unedited regions of the same audio clip.
• vs. TTT Literature (MAE-TTT, etc.): TTT has precedents in vision and NLP; this paper introduces it to speech editing for the first time.

Rating

• Novelty: ⭐⭐⭐⭐ First application of TTT to speech editing; the direct/indirect supervision design is clever.
• Experimental Thoroughness: ⭐⭐⭐ Limited by workshop paper scope, but both subjective and objective evaluations are included.
• Writing Quality: ⭐⭐⭐⭐ Problem motivation is clearly articulated.
• Value: ⭐⭐⭐⭐ Significant reference value for practical deployment of speech editing systems.

Related Papers

• [NeurIPS 2025] E-BATS: Efficient Backpropagation-Free Test-Time Adaptation for Speech Foundation Models
• [NeurIPS 2025] AVRobustBench: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time
• [ICCV 2025] Understanding Co-speech Gestures in-the-wild
• [NeurIPS 2025] VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
• [NeurIPS 2025] Node-Based Editing for Multimodal Generation of Text, Audio, Image, and Video