Skip to content

SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/daxintan-cuhk/SpeechEditBench
Area: Speech & Audio Processing
Keywords: Speech Editing, Speech LLM, Multi-Attribute Benchmark, Bilingual Evaluation, Compositional Editing

TL;DR

SpeechEditBench establishes the first bilingual, multi-attribute benchmark for instruction-guided speech editing covering seven atomic dimensions and multi-operation compositions, introducing an anchor-based evaluation protocol (target, preservation, and joint success) to systematically diagnose capability fragmentation and content preservation bottlenecks in modern Speech LLMs.

Background & Motivation

The recent surge of Speech Large Language Models (Speech LLMs) has catalyzed unified generative intelligence across speech recognition, text-to-speech synthesis, speech-to-speech translation, and spoken question answering. These classic speech processing capabilities are grounded in mature public datasets, standardized evaluation protocols, and competitive public leaderboards, which collectively drive continuous algorithmic refinement. However, instruction-guided speech editingโ€”a crucial capability demanding that a model dynamically modify targeted acoustic or linguistic properties in response to natural language promptsโ€”remains severely under-explored and fragmented across disconnected subfields.

The fundamental difficulty of speech editing stems from its inherent dual-constraint nature: an editing model must accurately execute modifications on designated attributes while strictly preserving all non-target characteristics, such as original textual wording, natural prosody, and speaker identity. Existing speech editing research and evaluation paradigms suffer from three major deficiencies. First, evaluation metrics are segregated across isolated task silos (e.g., inpainting vs. voice conversion) and cannot be meaningfully compared. Second, conventional speech evaluations over-rely on rigid waveform matching or reference-based distance metrics, failing to accommodate the inherently one-to-many valid acoustic realization space in speech editing. Third, existing studies almost exclusively assess target attribute modification success, ignoring unintended transcript drift or acoustic degradation in unedited segments.

To resolve this methodological impasse and deliver a rigorous diagnostic tool, this paper constructs SpeechEditBench, the first comprehensive bilingual benchmark for instruction-guided speech editing. SpeechEditBench unifies seven atomic editing operationsโ€”content, speaker, emotion, speaking style, prosody, paralinguistic events, and acoustic conditionsโ€”and further incorporates compositional multi-attribute tasks across English and Mandarin Chinese. Concurrently, it establishes an anchor-based evaluation framework that decouples target modification verification from non-target content preservation. Core idea: construct a comprehensive bilingual multi-attribute benchmark SpeechEditBench covering seven atomic tasks and compositional operations, supported by an anchor-based decoupled evaluation protocol (target, preservation, and joint success) to rigorously diagnose capability fragmentation and preservation bottlenecks across Speech LLMs.

Method

Overall Architecture

SpeechEditBench is built upon three foundational tenets: unified task formulation across heterogeneous speech dimensions, anchor-based decoupled evaluation to circumvent rigid waveform matching, and a dual-constraint metric system that balances edit effectiveness with source transcript preservation. The overall workflow consists of input formulation, hierarchical task definition, anchor generation, and multi-metric joint evaluation. An input sample pairs a source speech utterance with a natural language editing instruction (supplemented by a target reference audio clip for speaker editing). The evaluation engine extracts specific modification anchors and preservation anchors, using dedicated evaluators and an automatic speech recognition (ASR) preservation gate to determine target success \(t_i\) and preservation success \(p_i\), concluding with the joint success decision \(j_i = t_i \cdot p_i\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Formulation<br/>Source Audio + Text Instruction (+ Reference Speaker Audio)"] --> B["Atomic Task Taxonomy<br/>7 Core Dimensions: Content, Speaker, Emotion, Style, Prosody, Paralang, Acoustic"]
    B --> C["Compositional Task Construction<br/>Cross-Category 2-Component & 3-Component Combinations"]
    C --> D["Anchor-Based Evaluation Protocol<br/>Target Anchors (Edit Success) + Preservation Anchors (ASR Transcript Gate)"]
    D --> E["Joint Metric Decision System<br/>Target Success TS โˆง Preservation Success PS = Joint Success JS"]

Key Designs

1. Atomic Task Taxonomy: Multi-tiered coverage across semantic, vocal, and environmental dimensions

To overcome the narrow focus of previous editing datasets, SpeechEditBench structures the editing space into seven complementary atomic tasks spanning 4,300 carefully curated samples. At the semantic level, Content Editing comprises word/phrase replacement, insertion, and deletion (300 English from LibriTTS and 300 Chinese from AISHELL-3/WenetSpeech). At the speaker and delivery level, Speaker Editing handles voice conversion toward a reference identity (200 samples across VCTK and AISHELL-3); Emotion Editing features a standard neutral-text subset (650 samples) and a challenging subset (750 samples) where source text exhibits an inherent emotion conflicting with the target, testing whether models can override textual sentiment bias; Style Editing covers six distinct delivery personas (public-broadcast, intimate, dramatic, restrained-flat, storytelling, and conversational, totaling 600 samples); Prosody Editing manipulates speech rate, pitch shifts, or localized stress emphasis (600 samples); Paralinguistic Editing targets the insertion or removal of non-verbal vocal events including breathing, laughing, coughing, and sighing (400 samples); and Acoustic Editing spans speech enhancement (denoising/dereverberation) and environment transfer (injecting background noise or synthetic room impulse responses, 500 samples).

2. Compositional Task Construction: Diagnosing joint instruction execution under multi-constraint satisfaction

While models frequently excel at isolated single-attribute adjustments, practical human-AI voice interaction demands multi-attribute joint modifications within a single conversational turn. SpeechEditBench groups the atomic operations into four high-level categoriesโ€”semantic content, speaker identity, expressive delivery, and acoustic environmentโ€”and derives 400 compositional samples (320 two-component and 80 three-component combinations, precisely balanced between English and Chinese). Each compositional prompt operates on a single source speech utterance with atomic target anchors preserved for modular verification. For instance, a three-component instruction asks the model to simultaneously execute word replacement, speaker timbre transfer, and emotion conversion to frustrated. This protocol probes whether a model's latent representation can decouple and coordinate multiple acoustic-semantic shifts without mutual interference.

3. Anchor-Based Evaluation Protocol: Waveform-free assessment with hard linguistic preservation gating

To accommodate the intrinsic one-to-many realization of natural speech, the framework introduces anchor-based evaluation. For non-target linguistic preservation, the output is transcribed using Whisper large-v3 for English and Paraformer (via FunASR) for Chinese. After standard text normalization, Word Error Rate (WER) or Character Error Rate (CER) against the expected transcript is computed; a hard threshold \(e_i \le 0.10\) defines preservation success: \(p_i = \mathbb{I}[e_i \le 0.10]\). For target success \(t_i\), domain-tailored evaluators are deployed: content editing requires exact presence of target spans and absence of deleted spans; speaker editing requires cosine similarity \(\ge 0.50\) on WavLM-large + ECAPA-TDNN speaker embeddings; emotion, style, and paralinguistic edits are evaluated via structured zero-temperature Gemini 2.5 Pro audio judges; prosody editing enforces duration ratio bounds (\(\le 0.95\) for faster, \(\ge 1.05\) for slower), median F0 shifts (\(\ge +0.3\) or \(\le -0.3\) semitones), or timestamped word prominence gains; and acoustic editing mandates positive DNSMOS OVRL/BAK gains, RT60 matching within 0.8โ€“1.2ร— target ranges, or PANNs environmental scene classification gain \(\ge 0.03\).

4. Joint Metric Decision System: Exposing the target-preservation trade-off gap

Reporting only target fulfillment severely overstates model utility, as models can achieve acoustic targets by completely hallucinating or regenerating the audio. The protocol formalizes three foundational metricsโ€”Target Success (TS), Preservation Success (PS), and Joint Success (JS):

\[\text{TS} = \frac{1}{|D|}\sum_{i\in D} t_i, \quad \text{PS} = \frac{1}{|D|}\sum_{i\in D} p_i, \quad \text{JS} = \frac{1}{|D|}\sum_{i\in D} j_i \quad (j_i = t_i \cdot p_i)\]

For compositional evaluation, where a sample \(i\) entails a set of components \(C_i\) with individual successes \(a_{i,c} \in \{0, 1\}\), the benchmark measures pooled Component Success (CS), All-component Success (\(q_i = \prod_{c\in C_i} a_{i,c}\)), and Compositional Joint Success (\(j_i = q_i \cdot p_i\)). This formulation provides a rigorous penalty for outputs that satisfy target attributes at the expense of corrupting non-target content.

Key Experimental Results

Main Results

The benchmark systematically evaluates six open-source Speech LLMs (Ming-UniAudio, Step-Audio-EditX, Qwen3-Omni, Kimi-Audio, Mimo-Audio-Base, Mimo-Audio-Instruction), two closed-source commercial systems (Gemini-Live, GPT-Realtime), and five task-specific expert baselines (VoiceCraft-X, Seed-VC, VoxCPM2, Chatterbox+AudioSep, DeepFilterNet). Primary evaluation results on SpeechEditBench are compiled below (each cell reports Joint Success %, with Target Success % in parentheses; for compositional editing, parentheses show pooled Component Success %; content editing reports a single value as joint equals target):

Model Family Model Name Content โ†‘ Speaker โ†‘ Emotion โ†‘ Style โ†‘ Prosody โ†‘ Paralang โ†‘ Acoustic โ†‘ Compositional โ†‘
Open-source Speech LLMs Ming-UniAudio 76.46 N/T 3.43 (5.29) 22.17 (32.50) 26.50 (28.00) 11.25 (29.25) 25.85 (29.66) 1.75 (14.77)
Step-Audio-EditX 16.50 N/T 7.71 (9.29) 49.67 (54.00) 20.13 (51.51) 31.25 (61.75) 22.89 (40.96) 2.00 (16.14)
Qwen3-Omni 72.00 N/T 1.64 (15.29) 24.17 (63.50) 38.17 (44.83) 14.50 (44.75) 37.80 (42.00) 5.00 (31.59)
Kimi-Audio 34.67 N/T 2.36 (22.43) 24.50 (63.33) 13.50 (40.33) 9.25 (55.00) 8.25 (38.23) 1.50 (16.02)
Mimo-Audio-Base 31.67 N/T 0.21 (8.79) 5.83 (42.17) 5.67 (36.17) 1.00 (49.75) 4.44 (43.35) 1.75 (15.00)
Mimo-Audio-Inst. 64.17 N/T 0.86 (42.36) 0.50 (77.50) 0.67 (42.83) 2.00 (67.50) 0.80 (45.69) 7.25 (32.05)
Closed-source Speech LLMs Gemini-Live 93.17 N/T 27.79 (34.43) 63.67 (84.00) 65.17 (69.67) 26.50 (61.75) 36.69 (41.53) 11.00 (38.52)
GPT-Realtime 96.67 N/T 14.57 (21.00) 68.67 (82.33) 63.94 (70.12) 47.00 (81.50) 27.60 (43.60) 10.00 (34.89)
Specialized Systems Dedicated Baselines EN 84.00
ZH 47.00
80.50 (86.00) 3.57 (4.14) 32.67 (36.67) 49.00 (50.83) 19.25 (52.75) 68.20 (73.20) N/T

(Note: N/T denotes Not Tested; Speech LLM interfaces do not support standardized source-plus-reference audio input. Specialized baselines comprise VoiceCraft-X for content, Seed-VC for speaker, VoxCPM2 for emotion/style/prosody, Chatterbox+AudioSep for paralanguage, and DeepFilterNet/RIR for acoustic editing).

Ablation and Diagnostic Studies

1. Compositional Editing Breakdown (Speech-LLM-Compatible Subset):

Evaluating multi-intent editing capability by excluding reference-dependent speaker components shows a steep performance drop between isolated component execution and joint satisfaction (values in %):

Model Name 2-component: Comp. 2-component: All 2-component: Joint 3-component: Comp. 3-component: All 3-component: Joint
Ming-UniAudio 19.00 2.00 2.00 13.33 0.00 0.00
Step-Audio-EditX 22.00 4.00 3.50 23.33 0.00 0.00
Qwen3-Omni 40.50 10.50 10.00 43.33 0.00 0.00
Kimi-Audio 22.25 3.00 3.00 20.00 0.00 0.00
Mimo-Audio-Base 20.75 2.50 2.50 21.67 5.00 5.00
Mimo-Audio-Instruction 45.00 17.00 14.50 33.33 0.00 0.00
Gemini-Live 50.25 21.50 21.50 41.67 0.00 0.00
GPT-Realtime 49.75 20.50 20.00 40.00 0.00 0.00

2. Emotion Editing Under Lexical-Affective Conflict (Standard vs. Challenging):

The table below contrasts model resilience when source lexical content is emotionally neutral versus when it strongly conveys an emotion conflicting with the instructed target (\(\Delta = \text{Challenging} - \text{Standard}\) in percentage points):

Model Name TS: Standard TS: Challenging TS: \(\Delta\) JS: Standard JS: Challenging JS: \(\Delta\)
Ming-UniAudio 6.62 4.13 -2.49 5.08 2.00 -3.08
Step-Audio-EditX 8.62 9.87 +1.25 6.92 8.40 +1.48
Qwen3-Omni 14.15 16.27 +2.12 0.46 2.67 +2.21
Kimi-Audio 29.08 16.67 -12.41 3.69 1.20 -2.49
Mimo-Audio-Base 9.54 8.13 -1.41 0.15 0.27 +0.12
Mimo-Audio-Instruction 45.23 39.87 -5.36 0.46 1.20 +0.74
Gemini-Live 43.85 26.27 -17.58 37.08 19.73 -17.35
GPT-Realtime 22.77 19.47 -3.30 14.31 14.80 +0.49

Key Findings

  • Severe capability fragmentation across models: No single system achieves uniform competence. Ming-UniAudio leads open-source models in content editing (76.46% JS) but collapses on expressive tasks. Step-Audio-EditX excels at expressive style (49.67% JS) and paralanguage (31.25% JS), but fails in content preservation (16.50% JS). Qwen3-Omni achieves the top acoustic editing JS (37.80%), surpassing both proprietary models, yet achieves only 1.64% JS on emotion editing.
  • The target-preservation trade-off gap is prominent: Several models exhibit high target success but near-zero joint success due to catastrophic content drift. Mimo-Audio-Instruction achieves 77.50% TS in style, 67.50% TS in paralanguage, and 42.36% TS in emotion, but its corresponding joint success scores collapse to 0.50%, 2.00%, and 0.86% because it regularly regenerates or garbles non-target text tokens.
  • Compositional multi-intent editing remains an unsolved frontier: While leading models fulfill individual components at 40%โ€“50% rates in two-component tasks, joint success drops to 20%โ€“21.5% for Gemini-Live and GPT-Realtime, and 10% for Qwen3-Omni. In three-component compositions, all models except Mimo-Audio-Base (5.00%) plunge to a 0.00% joint success rate.
  • Asymmetric language bias: Models display distinct language preferences. In content editing, Ming-UniAudio and Step-Audio-EditX favor Mandarin (\(\Delta = +6.27\) and \(+29.67\)), whereas Gemini-Live and Kimi-Audio favor English (\(\Delta = -10.33\) and \(-29.33\)). In non-content tasks, almost all models preserve English text significantly better than Mandarin during acoustic/prosodic manipulation.

Highlights & Insights

  • Anchor-based decoupled evaluation solves the speech generation dilemma: Moving beyond rigid acoustic waveform matching, the framework separates target verification from ASR transcript gating, aligning automated speech evaluation with the natural one-to-many generative reality.
  • Joint Success serves as a crucial sanity check: Defining joint success as the logical conjunction of target modification and non-target preservation prevents models from gaming benchmarks by either ignoring instructions (perfect preservation, zero edit) or over-altering speech (target achieved, content destroyed).
  • Lexical-affective conflict subset isolates cross-modal cognitive barriers: The challenging emotion split reveals that models often suffer from semantic anchoring: when textual semantics contradict the target acoustic emotion, models struggle significantly more to assert acoustic control.

Limitations & Future Work

  • Linguistic and dialect breadth: The current benchmark is restricted to standard English and Mandarin Chinese, leaving out multi-lingual code-switching, low-resource languages, accented speech, and regional dialects.
  • Evaluator dependency and reproducibility: Qualitative assessment of emotion, style, and paralinguistic attributes relies on commercial Gemini 2.5 Pro APIs. While human validation confirms 68.8% agreement and strong correlation (\(r=0.685\)), proprietary API updates may affect longitudinal reproducibility.
  • Acoustic non-target preservation is not fully gated: The preservation gate currently enforces transcript fidelity via ASR WER/CER, but does not explicitly penalize drift in speaker identity or background prosody during non-speaker/non-prosody edits.
  • vs EditBench / EditVal / I2EBench (Vision): While vision benchmarks developed inpainting masks and VLM-based visual checks, SpeechEditBench brings this dual-objective discipline into the continuous time-domain acoustic realm, establishing principled modification and preservation anchors.
  • vs Ming-Freeform-Audio-Edit / Step-Audio-Edit-Benchmark: Earlier audio editing benchmarks used disjoint task definitions, lacked bilingual alignment, and omitted multi-attribute compositional testing. SpeechEditBench unifies seven atomic tasks under a coherent dual-constraint evaluation regime.
  • vs Specialized Expert Models (VoiceCraft-X, Seed-VC, DeepFilterNet): Specialized pipelines demonstrate high competency in dedicated domains (e.g., DeepFilterNet achieving 68.20% in acoustic editing), but cannot follow arbitrary natural language instructions. SpeechEditBench provides the roadmap for unifying these specialized strengths into general-purpose Speech LLMs.

Rating

  • Novelty: โญโญโญโญโญ Establishes the first unified bilingual multi-attribute benchmark for instruction-guided speech editing with anchor-based decoupled metrics.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluates 8 Speech LLMs and 5 specialized baselines across 4,700 samples, complemented by stratified human validation.
  • Writing Quality: โญโญโญโญโญ Rigorous task formulations, crisp mathematical formalization of evaluation criteria, and incisive diagnostic analysis.
  • Value: โญโญโญโญโญ Provides an indispensable diagnostic suite and standardized evaluation standard for the development of next-generation instruction-following Speech LLMs.