ARECHO: Autoregressive Evaluation via Chain-Based Hypothesis Optimization for Speech Multi-Metric Estimation

Conference: NeurIPS 2025 arXiv: 2505.24518 Code: https://github.com/ftshijt/espnet/tree/universa_plus Area: Interpretability Keywords: Speech multi-metric evaluation, autoregressive classification chain, tokenization, confidence-guided decoding, dependency modeling

TL;DR

ARECHO frames speech multi-metric evaluation as a chain-based autoregressive token prediction task. It designs a unified speech information tokenization pipeline to handle 87 heterogeneous metrics (numerical/categorical/bounded/unbounded), explicitly captures inter-metric dependencies (e.g., intelligibility–naturalness correlation) via dynamic classification chains, and employs two-step confidence-guided decoding to reduce error propagation. ARECHO comprehensively outperforms the UniVERSA baseline across enhancement, synthesis, and noisy speech evaluation (Avg Test MSE 23.26 vs. 96.99, −76%).

Background & Motivation

Background: Speech evaluation encompasses multiple metrics—PESQ (perceptual quality), STOI (intelligibility), MOS (subjective scores), speaker similarity, emotion recognition, etc. Unified frameworks such as UniVERSA and TorchSquim support multi-metric prediction.

Limitations of Prior Work: (a) Scale heterogeneity—MOS ranges from 1 to 5 while SI-SNR ranges over \((-\infty, +\infty)\); a shared L1 loss causes large-range metrics to dominate optimization. (b) Inter-metric dependencies ignored—improvements in intelligibility typically co-occur with improvements in naturalness, yet parallel prediction cannot exploit such correlations. (c) Partial annotation—PESQ requires a clean reference, WER requires transcripts, so real-world data are often only partially annotated.

Key Challenge: Parallel prediction is efficient but discards inter-metric dependency information; modeling dependencies, however, introduces the triple challenges of heterogeneous scales, partial annotation, and error propagation.

Goal: Design a dependency-aware multi-metric evaluation framework that simultaneously handles scale heterogeneity and partial annotation.

Key Insight: Tokenize all metrics into a unified discrete space—quantizing numerical metrics into bin tokens and mapping categorical metrics directly to tokens—then model conditional inter-metric dependencies via autoregressive classification chains.

Core Idea: Unified tokenization of 87 heterogeneous metrics → dynamic classification chain autoregressive prediction (conditioning subsequent predictions on previously predicted metrics) → two-step confidence-guided decoding to reduce error propagation = dependency-aware speech multi-metric evaluation.

Method

Overall Architecture

Speech signal \(\mathbf{S}\) → WavLM audio encoder → shared representation → Tokenization: all 87 metrics mapped to discrete tokens \(Z = \{z_b\}_{b \in \mathcal{B}}\) → Dynamic classification chain: interleaved sequence \(\mathbf{T} = [m_{b_1}, z_{b_1}, m_{b_2}, z_{b_2}, \ldots]\) (alternating metadata tokens and value tokens) → Transformer decoder autoregressive generation → Two-step confidence-guided decoding: for each metric to be predicted, Top-B candidate values are evaluated and the one with highest log-likelihood is retained.

Key Designs

1. Unified Speech Information Tokenization:

   • Function: Map 65 numerical metrics and 22 categorical metrics into a unified discrete space.
   • Mechanism: Numerical metrics are quantized into \(B\) bins (linear/normal/log schemes to accommodate different distributions); categorical metrics are mapped directly to tokens. The inverse mapping \(\mathcal{T}_b^{-1}\) recovers predicted values. The total vocabulary is \(\mathcal{V} = \bigcup_b \mathcal{V}_b\).
   • Design Motivation: Tokenization eliminates scale discrepancies by casting all metrics as classification problems in the same discrete space. Ablation shows tokenization alone yields substantial gains—UniVERSA-T (MSE 37.72) vs. UniVERSA (96.99).

2. Dynamic Classification Chain:

   • Function: Predict metrics autoregressively, conditioning each prediction on previously predicted metrics.
   • Mechanism: The target sequence is \(\mathbf{T} = [m_{b_1}, z_{b_1}, m_{b_2}, z_{b_2}, \ldots]\), where metadata token \(m_b\) indicates which metric to predict next and value token \(z_b\) is the prediction. During training, metric order is randomly shuffled so the model learns arbitrary conditioning patterns. Missing metrics are simply omitted for partially annotated data, naturally supporting weak supervision.
   • Design Motivation: Random-order training enables the model to query metrics in any order or over any subset at inference time, providing maximal flexibility.

3. Two-Step Confidence-Guided Decoding:

   • Function: Reduce error propagation in autoregressive prediction.
   • Mechanism: For each remaining metric \(b\)—Step 1: append metadata token \(m_b\) and autoregressively generate a tentative value token \(\hat{z}_b\) with confidence \(\text{Conf}(\hat{z}_b)\). Step 2: try Top-B candidate values \(\tilde{z}_b\) after \(m_b\), compute the sequence log-likelihood for each candidate, and retain the best. After comparing candidates across all metrics, the Top-B partial hypotheses are kept as the prefix for the next step.
   • Design Motivation: Confidence estimates for metadata tokens are unreliable due to random order training; value token sequence likelihoods are therefore used for guidance—analogous to beam search but operating in the metric dimension.

Loss & Training

• Autoregressive cross-entropy loss: \(P(\mathbf{T}^i | \mathbf{S}^i) = \prod_t P(x_t | \mathbf{T}_{<t}, \mathbf{S})\)
• The VERSA toolkit automatically computes 87 metrics (47 independent + 25 dependent + 7 non-matching + 8 annotated).
• Base training set: 308.77 hours; Scale training set: 2,137.74 hours.
• Architecture: WavLM encoder + Transformer decoder.

Key Experimental Results

Main Results (Base Training, Avg Test)

Model	Tokenization	Chain	MSE↓	LCC↑	KTAU↑	Acc↑	F1↑
UniVERSA	✗	✗	96.99	0.69	0.52	0.69	0.45
UniVERSA-T	✓	✗	37.72	0.79	0.68	0.71	0.49
ARECHO	✓	✓	23.26	0.86	0.72	0.74	0.57

Per-Scenario Comparison (Base Training)

Scenario	ARECHO MSE	UniVERSA MSE	Reduction
Enhanced speech	20.58	61.54	−67%
Noisy speech	44.22	170.65	−74%
Synthesized speech	4.99	58.79	−91%
Dev set	25.73	160.06	−84%

Scale Training (2,137 hours)

Model	Avg Test MSE	LCC	F1
UniVERSA	67.16	—	—
UniVERSA-T	—	—	—
ARECHO	Improved	—	—

Key Findings

• Tokenization is the single largest source of improvement: MSE drops from 96.99 (UniVERSA) to 37.72 (UniVERSA-T)—a 61% reduction from tokenization alone, confirming that scale disparity is a critical bottleneck.
• The classification chain yields additional gains on top of tokenization: 37.72 → 23.26 (a further 38% reduction), demonstrating that dependency modeling provides independent value.
• Synthesized speech shows the largest improvement (−91%): likely because inter-metric correlations are strongest for synthesized speech (e.g., naturalness and MOS are highly correlated), which the classification chain fully exploits.
• Confidence-guided decoding effectively reduces error propagation: the benefit is most pronounced in long-chain prediction with many metrics.
• Partial annotation is natively supported: missing metrics are simply omitted without any special handling.

Highlights & Insights

• Tokenization is an underappreciated technique: unifying heterogeneous numerical regression into classification is conceptually simple yet remarkably effective (MSE −61%), suggesting that scale mismatch is a more severe problem than previously recognized.
• The dynamic classification chain is highly flexible: random-order training enables arbitrary subset/order queries at inference with a single model, accommodating all evaluation scenarios.
• Two-step confidence-guided decoding is beam search in the metric dimension: it transfers token-level beam search from NLP to the metric level in a natural and effective manner.
• Unified modeling of 87 metrics represents the most comprehensive speech evaluation framework to date, covering quality, intelligibility, speaker characteristics, emotion, and environmental dimensions.

Limitations & Future Work

• Autoregressive prediction is slower than parallel prediction; chain-based decoding over 87 metrics requires multiple inference steps.
• Chain order affects results; although random-order training mitigates this, the optimal ordering remains unknown.
• Tokenization introduces quantization error; the number of bins requires balancing precision against classification difficulty.
• The advantage of ARECHO over UniVERSA diminishes under Scale training, suggesting that large data may partially compensate for the information loss in parallel prediction.
• Multi-modal inputs (e.g., reference audio, text transcripts) are not currently supported and are left for future work.

Related Work & Insights

• vs. UniVERSA: Parallel metric prediction ignores inter-metric dependencies; ARECHO models them explicitly via classification chains—MSE −76%.
• vs. TorchSquim: A similar parallel framework but with fewer metrics; ARECHO covers 87 metrics.
• vs. LLM-based speech evaluation: Natural language quality descriptions are less precise than ARECHO's token-level metric modeling for exact scoring.
• vs. Classifier Chains: A classical multi-label method; ARECHO extends it to the token level with dynamic ordering and partial annotation support.
• Insight: The tokenization + autoregressive chain paradigm is transferable to any multi-metric prediction task, such as multi-dimensional image quality assessment or multi-metric medical diagnostic reporting.

Rating

• Novelty: ⭐⭐⭐⭐ Triple innovation of tokenization + dynamic classification chain + confidence-guided decoding.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Three scenarios, 87 metrics, Base/Scale training regimes, and thorough ablation studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear challenge–solution correspondence; complete mathematical derivations.
• Value: ⭐⭐⭐⭐⭐ The most comprehensive speech evaluation framework to date, with significant contributions to the speech/audio community.

Related Papers

• [NeurIPS 2025] Better Estimation of the Kullback-Leibler Divergence Between Language Models
• [AAAI 2026] Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
• [NeurIPS 2025] AgentiQL: An Agent-Inspired Multi-Expert Framework for Text-to-SQL Generation
• [NeurIPS 2025] Sloth: Scaling Laws for LLM Skills to Predict Multi-Benchmark Performance Across Families
• [ACL 2026] TabReX: Tabular Referenceless eXplainable Evaluation