Why We Need Speech to Evaluate Speech Translation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/MaikeZuefle/speechCOMET
Area: Audio & Speech
Keywords: speech translation evaluation, quality estimation, gender agreement, prosody, multimodal representations
TL;DR¶
This paper benchmarks text- and speech-aware quality estimation (QE) metrics on speech-specific phenomena, trains the SpeechCOMET family and evaluates SpeechLLMs as judges, revealing that existing metrics perform at chance level on gender and prosodic distinctions despite speech input.
Background & Motivation¶
Speech translation (ST) models are rapidly advancing from cascaded pipelines to end-to-end architectures and multimodal SpeechLLMs that process audio signals directly. This transition enables systems to theoretically capture and preserve speech-specific nuances that have no textual footprint in the source utteranceโsuch as the speaker's vocal gender, prosodic phrasing, question intonation, lexical stress, and affective cues. In real-world spoken interactions, identical lexical transcripts often carry completely distinct translations depending on acoustic conditioning: prosodic breaks disambiguate syntactic attachments, and grammatical gender inflections in target languages such as Spanish or Italian strictly require discerning the speaker's voice.
Despite this architectural maturation, evaluation methodology remains predominantly trapped in text-based paradigms. Standard speech translation evaluations continue to rely on reference-free quality estimation (QE) models like WMT22 COMETKiwi applied to human transcripts or automatic speech recognition (ASR) outputs. Relying on ASR transcripts introduces transcription error cascades, whereas relying on text transcripts by construction blinds the metric to all acoustic paralinguistic phenomena. Although multimodal, speech-aware metrics such as BLASER 2.0 and SpeechQE have been introduced, they have historically been validated solely via coarse correlation against human direct assessments on general sentence datasets, leaving completely unexamined whether they actually condition on or evaluate speech-specific properties.
This divergence creates a critical blind spot across the field: practitioners lack metrics capable of auditing whether modern ST models faithfully convey speech-grounded meaning, and existing speech-fed metrics risk operating as ungrounded black boxes that judge only target surface fluency. Core idea: meta-evaluate existing text and speech metrics alongside newly trained SpeechCOMET models and fine-tuned SpeechLLM judges on dedicated contrastive benchmarks, demonstrating that all current metrics fail at chance level on gender agreement and prosody due to acoustic representation erasure, lack of source conditioning, and insufficient speech-specific training supervision.
Method¶
Overall Architecture¶
The evaluation framework establishes an end-to-end meta-evaluation and diagnostic pipeline spanning contrastive benchmark curation, multi-encoder metric modeling, large-model judging, and multi-layer representation probing. Given paired source speech, corresponding transcript (reference or ASR), and translation hypotheses, the pipeline evaluates both continuous score correlation on general quality benchmarks and pairwise accuracy on fine-grained acoustic contrastive sets, followed by probing and source shuffling to pinpoint systemic points of failure.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
IN["Source Input: Audio Waveform / Text Transcript + MT Hypothesis"] --> D1["Dual-Track Contrastive Meta-Evaluation<br/>IWSLT 2026 / MuST-SHE / ContraProST"]
D1 --> D2["SpeechCOMET Quality Estimation Modeling<br/>SONAR / Whisper Speech Encoders + Multimodal Fusion"]
D2 --> D3["SpeechLLM Multimodal Judge Paradigm<br/>Qwen2.5-Omni Zero-Shot & LoRA Fine-Tuning"]
D3 --> D4["Multi-Dimensional Failure Attribution Analysis<br/>Linguistic/Acoustic Probing & Source Shuffling Ablations"]
D4 --> OUT["Benchmark Finding: Metrics Degrade to Chance on Speech Nuances"]
Key Designs¶
1. Dual-Track Contrastive Meta-Evaluation: Unmasking Paralinguistic Blind Spots Behind Global Correlation
Standard QE benchmarks evaluate system predictions against human direct assessment scores using segment-level Kendall's \(\tau_b\) and system-level Soft Pairwise Accuracy (SPA). However, global correlation is dominated by general semantic adequacy and length heuristics, which easily masks catastrophic failures on sparse acoustic cues. To isolate speech-specific sensitivity, this work employs two minimally contrastive benchmarks where translations can only be verified by listening to the source audio: MuST-SHE (1,612 contrastive pairs for Enโ{Es, Fr, It} testing speaker gender agreement in grammatical inflections) and ContraProST (15,972 contrastive pairs across Enโ{De, Es, Ja} assessing sentence stress, intonation, prosodic breaks, emotional prosody, and politeness). Evaluation is quantified via Pairwise Accuracy (PA): $$ \text{PA} = \frac{1}{N} \sum_{k=1}^N \mathbf{1}[\hat{y}_k^+ > \hat{y}_k^-] $$ where \(\hat{y}_k^+\) and \(\hat{y}_k^-\) denote the predicted quality scores for the correct and incorrect contrastive translations, respectively. A model operating without acoustic grounding scores around the 50% random-choice baseline, with ties counted as incorrect.
2. SpeechCOMET Quality Estimation Modeling: Speech Encoder Integration and Multimodal Source Fusion
To test whether the absence of speech sensitivity in existing metrics stems from architectural deficits, the authors design and train the SpeechCOMET family. While text COMETKiwi extracts source vector \(s_t \in \mathbb{R}^d\) and hypothesis vector \(h_t \in \mathbb{R}^d\) through a shared multilingual encoder before feeding the four-way interaction \(e = [h_t; s_t; |h_t - s_t|; h_t \odot s_t] \in \mathbb{R}^{4d}\) into a two-layer Tanh MLP regression head, SpeechCOMET replaces \(s_t\) with a speech embedding \(s_a \in \mathbb{R}^d\) via a learned linear projection. The speech representations are extracted using either SONAR (a multilingual sentence-level speech encoder) or Whisper Large V3 (a frame-level ASR encoder combined with average or attention-based temporal pooling). Furthermore, SpeechCOMET extends to multimodal dual-source inputs by fusing text and speech representations via element-wise summation \(s = s_t + s_a\). Initializing the multimodal model from a text-pretrained checkpoint trained on large-scale WMT data provides substantial transfer gains on standard IWSLT metrics.
3. SpeechLLM Multimodal Judge Paradigm: Zero-Shot Paralinguistic Prompting and LoRA Supervised Tuning
Investigating generative multimodal judges, the authors adapt Qwen2.5-Omni (7B), a unified model supporting text, audio, and interleaved audio-text inputs. In the zero-shot regime, the model is queried using standard QE prompts alongside specialized task prompts: a gender-aware prompt directing the model to attend to vocal pitch and timbre for grammatical gender agreement, and a prosody-aware prompt instructing it to evaluate rhythm, pauses, and stress. In the fine-tuned regime, LoRA adapters (rank \(r=16, \alpha=32\)) are optimized across attention and MLP modules on the IWSLT human-annotated speech translation dataset using LlamaFactory. This evaluates whether large-scale multimodal pretraining can resolve acoustic evaluation challenges where discriminative estimators struggle.
4. Multi-Dimensional Failure Attribution Analysis: Feature Probing and Source Shuffling Ablations
To systematically identify why speech-fed models fail on acoustic phenomena, the authors formulate two diagnostic protocols. First, linear and MLP probing classifiers are trained on frozen encoder states to verify whether targeted properties survive representation extraction: five linguistic features (subject/object number, tense, verb form, named entities) are probed on IWSLT, while three acoustic features (speaker gender, question/statement intonation, emotion) are probed on MuST-SHE and ContraProST. Second, source reliance is quantified by replacing true source inputs (speech, text, or both) with randomly mismatched source samples from the corpus at test time, measuring the drop \(\Delta_{\text{random}} = \tau_b^{\text{random}} - \tau_b^{\text{real}}\). Statistically insignificant drops directly identify models that disregard the source input and rely exclusively on target-side prior heuristics.
Key Experimental Results¶
Main Results¶
The table below reports meta-evaluation results on the IWSLT 2026 Metrics development set (en-de and en-zh segments with human direct assessments) and contrastive pairwise accuracy (PA, %) on MuST-SHE and ContraProST. Segment-level correlation is reported in Kendall's \(\tau_b\) (%), system-level ranking in Soft Pairwise Accuracy (SPA, %), and ASR degradation represents \(\Delta\tau_b\) when replacing human transcripts with Whisper Large V3 ASR text.
| Model Category | Model Name | IWSLT Seg. \(\tau_b\) | IWSLT Sys. SPA | ASR Drop \(\Delta\tau_b\) | MuST-SHE PA (es/fr/it) | ContraProST PA (de/es/ja) |
|---|---|---|---|---|---|---|
| Text Baseline | COMET-PARTIAL | 11.9 | 55.0 | -0.3 | 52.4 / 55.7 / 51.8 | 50.0 / 50.0 / 50.0 |
| Text Baseline | COMETKiwi (WMT22) | 34.6 | 87.7 | -6.0 | 61.5 / 60.6 / 55.2 | 50.0 / 50.0 / 50.0 |
| Speech Baseline | SpeechQE | 29.8 | 75.2 | โ | 37.0 / 36.0 / 31.0 | 26.1 / 20.0 / 38.2 |
| Speech Baseline | BLASER 2.0 | 24.4 | 76.5 | โ | 52.0 / 51.6 / 51.7 | 51.0 / 51.5 / 51.1 |
| SpeechCOMET | COMETKiwi (IWSLT-RoBERTa) | 22.8 | 68.7 | -0.5 | 52.8 / 49.1 / 52.0 | 50.0 / 50.0 / 50.0 |
| SpeechCOMET | SpeechCOMET (SONAR) | 20.1 | 61.0 | โ | 53.1 / 50.9 / 50.9 | 50.0 / 50.0 / 50.0 |
| SpeechCOMET | SpeechCOMET (Whisper) | 17.8 | 56.7 | โ | 51.4 / 50.7 / 52.8 | 50.0 / 50.0 / 50.1 |
| SpeechCOMET | SpeechCOMET (Speech+Text) | 25.8 | 73.5 | -3.8 | 55.6 / 52.6 / 52.4 | 50.0 / 49.9 / 50.0 |
| SpeechCOMET | SpeechCOMET-XL (Speech+Text) | 34.4 | 76.5 | -6.0 | 54.6 / 58.0 / 51.1 | 50.0 / 50.0 / 50.0 |
| SpeechLLM | Qwen2.5-Omni (Text) | 40.4 | 64.1 | -5.5 | 23.7 / 17.3 / 16.2 | 23.8 / 18.1 / 31.1 |
| SpeechLLM | Qwen2.5-Omni (Text+FT) | 47.1 | 88.5 | -3.8 | 14.0 / 13.8 / 15.4 | 14.0 / 14.0 / 26.1 |
| SpeechLLM | Qwen2.5-Omni (Speech) | 32.0 | 60.1 | โ | 38.0 / 32.3 / 32.4 | 32.2 / 28.4 / 33.1 |
| SpeechLLM | Qwen2.5-Omni (Speech+FT) | 40.1 | 70.1 | โ | 35.9 / 15.0 / 20.7 | 21.1 / 21.9 / 15.7 |
| SpeechLLM | Qwen2.5-Omni (Speech+Text) | 38.1 | 61.6 | -4.5 | 43.1 / 37.1 / 37.8 | 31.0 / 26.1 / 32.6 |
| SpeechLLM | Qwen2.5-Omni (Speech+Text+FT) | 49.9 | 80.6 | -4.6 | 8.2 / 6.3 / 6.8 | 17.8 / 8.8 / 20.4 |
Ablation Study¶
The table below details the random source mismatch ablation (measuring degree of source conditioning) alongside source representation fusion ablations for SpeechCOMET:
| Experimental Setting | Configuration | Real Source \(\tau_b\) (%) | Shuffled Drop \(\Delta_{\text{random}}\) (pp) | Analytical Finding |
|---|---|---|---|---|
| Source Reliance Ablation | COMETKiwi (Text) | 34.6 | -24.9 | Strong reliance on textual source signal |
| Source Reliance Ablation | SpeechQE (Speech) | 29.8 | -22.5 | Conditions on speech, but insensitive to fine nuances |
| Source Reliance Ablation | SpeechCOMET (SONAR Speech-only) | 20.1 | -3.1 | Statistically insignificant drop; ignores source audio |
| Source Reliance Ablation | SpeechCOMET (Whisper Speech-only) | 17.8 | -2.4 | Statistically insignificant drop; ignores source audio |
| Source Reliance Ablation | SpeechCOMET (Speech+Text: Shuffled Audio) | 25.8 | -1.4 | Audio shuffling produces virtually no performance change |
| Source Reliance Ablation | SpeechCOMET (Speech+Text: Shuffled Text) | 25.8 | -17.2 | Text shuffling reproduces nearly full multimodal drop (-17.8) |
| Multimodal Fusion Ablation | Element-wise Sum (Frozen SONAR) | 22.8 | โ | Summation delivers best segment-level balance |
| Multimodal Fusion Ablation | Concatenation + Linear Projection | 16.6 | โ | Projection of concatenated inputs severely underfits |
| Multimodal Fusion Ablation | Average Pooling Fusion | 20.8 | โ | Simple averaging underperforms element-wise summation |
| Multimodal Fusion Ablation | Text Checkpoint Init + Sum FT | 25.8 | โ | Pretraining on WMT text QE provides +3.0 pp gain |
Key Findings¶
- Complete Failure on Speech-Specific Benchmarks: While fine-tuned SpeechLLM reaches an impressive 49.9% \(\tau_b\) and SpeechCOMET-XL matches text-based COMETKiwi (34.4% vs 34.6%) on standard IWSLT evaluation, all regression models hover precisely around 50% random chance on MuST-SHE and ContraProST. Generative models (SpeechLLM, SpeechQE) score well below 50% due to pervasive tie predictions where both options receive identical scores.
- Acoustic Information Bottleneck in Encoders: Probing demonstrates that SONAR's cross-lingual semantic alignment objective discards acoustic variance, rendering speaker gender and emotion near majority-baseline decodability. Conversely, Whisper and SpeechLLM representations retain rich speaker gender and intonation cues, but downstream quality estimation objectives fail to leverage them.
- Severe Text Dominance and Audio Neglect: In dual-input setups, models develop an overwhelming shortcut bias toward text. Shuffling the source audio causes only a -1.4 pp drop in SpeechCOMET, whereas shuffling the source text collapses performance by -17.2 pp, proving the audio stream is effectively bypassed during evaluation.
Highlights & Insights¶
- Exposes a foundational vulnerability in speech translation evaluation by empirically proving that current speech-aware metrics are completely insensitive to vocal gender, stress, and intonation.
- Establishes a rigorous diagnostic paradigm combining contrastive unit testing, representation probing, and source shuffling to expose modality bypassing in multimodal estimators.
- Provides actionable recommendations: practitioners should adopt fine-tuned SpeechLLMs or COMETKiwi for general translation adequacy, while refraining from using any current metric to assess paralinguistic fidelity until acoustically grounded training sets are developed.
Limitations & Future Work¶
- Meta-evaluations are anchored primarily to the English-source IWSLT 2026 dev set, leaving non-English and low-resource source speech behaviors unmeasured.
- Paralinguistic testing covers gender and prosody but omits dialectal variation, speaker accent, background acoustic noise, and conversational turn-taking dynamics.
- Synthetic TTS training data augmentation (Kokoro TTS on WMT) fails to instill sensitivity to natural human vocal nuances, highlighting the need for curated human contrastive QE corpora.
Related Work & Insights¶
- vs COMETKiwi: COMETKiwi remains the dominant text QE metric but drops by 6.0 pp under ASR input and cannot evaluate vocal nuances. SpeechCOMET matches its adequacy correlation when scaled, but inherits the inability to resolve speech-specific phenomena.
- vs BLASER 2.0 / SpeechQE: While BLASER utilizes shared SONAR embeddings and SpeechQE cascades Whisper into an LLM, both fail to surpass chance on MuST-SHE and ContraProST despite having direct access to audio waveforms.
Rating¶
- Novelty: โญโญโญโญโญ Conducts the first comprehensive meta-evaluation of speech translation QE on speech-specific phenomena, debunking existing assumptions.
- Experimental Thoroughness: โญโญโญโญโญ Rigorously benchmarks 15+ model setups across contrastive datasets, multi-layer probing, and input-shuffling ablations.
- Writing Quality: โญโญโญโญโญ Exceptionally clear narrative progression connecting empirical failures to concrete architectural and training causes.
- Value: โญโญโญโญโญ Critical roadmap for speech and multimodal translation communities, defining necessary prerequisites for next-generation evaluation.