AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/CuCl-2/AnyAudio-Judge
Area: LLM Reasoning
Keywords: Audio Instruction Following, Dynamic Rubric, Large Audio Language Models, Reinforcement Learning, Alignment Evaluation
TL;DR¶
AnyAudio-Judge establishes a dynamic rubric-based evaluation paradigm that decomposes complex audio instructions into independent, verifiable binary rubric items and aggregates soft satisfaction probabilities, supported by a 7,920-sample bilingual hard-negative benchmark (AnyAudio-Judge Bench) and a 105K CoT training corpus, where the resulting SFT+GRPO-tuned evaluator achieves state-of-the-art alignment detection and provides dense reward signals for downstream InstructTTS reinforcement learning.
Background & Motivation¶
Recent breakthroughs in diffusion models and multimodal architectures have catalyzed substantial progress in instruction-guided audio generation, spanning zero-shot expressive text-to-speech (InstructTTS), open-domain text-to-audio sound synthesis (TTA), and controllable music composition (TTM). Modern foundation audio models are increasingly expected to adhere to complex, open-ended textual prompts that simultaneously govern linguistic semantic content, speaker identity, emotional nuances, subtle background acoustic layers, and the precise chronological ordering of acoustic events. Consequently, accurately evaluating whether the synthesized audio faithfully aligns with such multi-faceted, highly constrained instructions has emerged as a critical bottleneck for the audio synthesis and alignment community.
Traditional evaluation paradigms have historically relied on either global continuous objective metrics (such as CLAP cosine similarity embeddings) or costly human subjective assessments. While CLAP representations offer a coarse estimate of holistic semantic proximity, they suffer from inherent blind spots when diagnosing fine-grained local attribute mismatches, such as secondary sound omissions, temporal sequence inversion, or subtle tonal contradictions. Conversely, human evaluation is prohibitively expensive, non-scalable, and vulnerable to subjective listener biases. To address these limitations, recent efforts have increasingly adopted Large Audio Language Models (LALMs) or proprietary multimodal LLMs (e.g., Gemini, GPT-4) as automated surrogate judges. Nonetheless, the vast majority of existing evaluators treat alignment judgment as a monolithic black-box task, prompting models to deliver a single, coarse-grained "yes/no" or scalar verdict. This holistic approach fails to decouple intricate compositional prompts, frequently halluncinates on long-tail constraints, and completely lacks the interpretability required to identify exactly which requested acoustic attribute failed during synthesis.
Compounding this problem, the research landscape has lacked comprehensive, multi-domain benchmarks specifically constructed with targeted, high-confusion "hard negatives" to rigorously probe the discriminative boundaries of audio evaluators. Core idea: reconstruct monolithic black-box audio evaluation into a dynamic rubric-based verification paradigm, where complex audio instructions are adaptively decomposed into a variable set of atomic, verifiable binary rubric items whose logit-derived satisfaction probabilities are aggregated into a transparent alignment score, and train a dedicated evaluator over a 105K CoT corpus via SFT and GRPO that doubles as an effective dense reward model for generative audio reinforcement learning.
Method¶
Overall Architecture¶
The AnyAudio-Judge framework operates through a decoupled two-stage evaluation pipeline complemented by a scalable data construction and policy optimization loop. Given an audio-instruction pair \((a, i)\), the framework avoids issuing an immediate monolithic decision. Instead, Stage 1 deploys an LLM (such as Qwen3-30B-A3B-Instruct) to decompose the complex instruction \(i\) into \(n\) independent, directly perceivable, and verifiable binary rubric questions \(\{p_1, p_2, \dots, p_n\}\). In Stage 2, the dedicated audio evaluator model, AnyAudio-Judge, processes the audio alongside each individual rubric item \(p_j\), extracting candidate token logits to compute a soft satisfaction probability \(p_j^{\text{yes}}\), which is then aggregated across all \(n\) items into an overall alignment score \(s\). To endow the model with robust auditory reasoning and grounded verification capabilities, the authors construct the 105K-sample AnyAudio-Judge Corpus and apply a two-stage training paradigm combining Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Pair (Audio a, Instruction i)"] --> B["Dynamic Rubric Decomposition<br/>LLM adaptively generates n atomic items"]
B --> C["Binary Soft Satisfaction Evaluation<br/>Extract yes/no logits & softmax average"]
C --> D["Bilingual Hard-Negative Benchmark<br/>Mining failures + attribute perturbations"]
D --> E["Two-Stage Evaluator Alignment<br/>105K SFT + 8.4K hard-sample GRPO"]
E --> F["Downstream Reinforcement Learning<br/>Dense reward feedback for InstructTTS"]
Key Designs¶
1. Dynamic Rubric Decomposition: Adaptive Instruction Decoupling into Atomic Verifiable Items To resolve the vulnerability of holistic evaluators when processing composite, constraint-heavy instructions, this module uses structured LLM prompting (Qwen3-30B-A3B-Instruct-2507) to unpack captions into atomic binary questions. The decomposition enforces five foundational principles: atomicity (each item isolates exactly one speaker, dimension, or feature), verifiability (each item checks only objective, audibly discernable acoustic features rather than subjective impressions), faithfulness (grounded strictly in explicit or strongly implied caption content), completeness (covering all primary components and spatial/temporal mix relations), and objectivity. The number of generated rubric items \(n\) scales dynamically with prompt complexity, exhibiting a strong linear correlation with instruction word count (fitted slope of 0.1258). Crucially, all questions are strictly normalized such that answering "yes" unambiguously signifies alignment with the original caption; negative constraints (e.g., "no obvious reverb") are rephrased as "Is there no obvious reverberation?" to prevent polarity reversal. A secondary LLM auditor removes any hallucinated or unsupported rubric questions before audio evaluation.
2. Binary Soft Satisfaction Evaluation: Logit-Level Confidence Aggregation and Attribute Attribution To overcome the coarse discreteness of standard binary outputs and provide a smooth, continuous optimization landscape, the evaluator model assesses each rubric item \(p_j\) by directly examining the unnormalized log-probabilities (logits) assigned to candidate tokens "yes" (\(z_j^{\text{yes}}\)) and "no" (\(z_j^{\text{no}}\)). Applying a two-way softmax produces the soft satisfaction probability: $\(p_j^{\text{yes}} = \frac{\exp(z_j^{\text{yes}})}{\exp(z_j^{\text{yes}}) + \exp(z_j^{\text{no}})}\)$ The overall cross-modal semantic alignment score \(s\) is computed as the unweighted arithmetic mean over all \(n\) decomposed rubric items: $\(s = \frac{1}{n} \sum_{j=1}^n p_j^{\text{yes}}\)$ This continuous score \(s \in [0, 1]\) not only pinpoints precisely which acoustic attribute (e.g., emotional intensity, background layer, dialect accent) failed to materialize in the audio, but also supplies a fine-grained, low-variance supervisory reward signal suitable for downstream reinforcement learning.
3. Bilingual Hard-Negative Benchmark: Generator Failure Mining and Systematic Attribute Perturbation To rigorously evaluate judge models on subtle acoustic discrepancies rather than trivial low-level mismatches, the authors construct the AnyAudio-Judge Bench, comprising 7,920 balanced samples across speech, environmental sound, music, and mixed cinematic audio. Negative sample generation adopts a dual strategy: (1) True Generator Failure Mining: Generating audio via cutting-edge systems (Qwen3-TTS, MOSS-VoiceGenerator, AudioGen, MusicGen) and deploying dual-pass Gemini verification to retain genuine alignment synthesis failures; and (2) Controlled Attribute Perturbation & Swapping: Applying systematic perturbations via LLMs to alter subtle acoustic dimensions while preserving scene plausibility. In speech, perturbations target dynamic trajectory reversal, emotional intensity modulation, terminal intonation shift, accent/dialect substitution, and pitch/age alterations; in sound, perturbations simulate secondary sound omission, temporal shuffling, and acoustic space enrichment; in music, perturbations induce instrument substitutions, tempo mismatches, and background layer drops. The entire benchmark is mirrored into a fully symmetric English and Chinese parallel set by translating textual instructions while keeping raw audio identical, establishing a rigorous testbed for cross-lingual robustness.
4. Two-Stage Evaluator Alignment: SFT Cold-Start and GRPO Optimization on Hard Ambiguities To elevate performance beyond prompt-engineered off-the-shelf LALMs, a dedicated evaluator is trained on the 105K-sample AnyAudio-Judge Corpus (spanning 30K Speech, 30K Sound, 30K Music, and 15K Mixed samples with full CoT rationale supervision). Starting from Qwen3-Omni-30B-A3B-Captioner, Stage 1 performs full-parameter SFT, conditioning the model to output structured JSON arrays containing rubric IDs, binary answers, and concise hearing evidence. In Stage 2, the authors perform four rollout passes over the corpus to filter out consistently solved instances, isolating 8,454 hard, ambiguous samples for Group Relative Policy Optimization (GRPO). LoRA adaptation (rank 16, alpha 32) is applied with a composite reward function balancing three complementary objectives: format consistency (weight 0.1, ensuring JSON schema integrity), global accuracy (weight 0.2, penalizing any sample marked match when a rubric item failed), and balanced rubric accuracy (weight 0.7, computing the average accuracy across gold-yes and gold-no items to prevent mode collapse into trivial majority-class prediction).
Loss & Training¶
AnyAudio-Judge was trained on 16 NVIDIA H200 (96GB) GPUs. SFT utilized full-parameter fine-tuning for 1 epoch with a per-device batch size of 4, gradient accumulation of 1, and a learning rate of \(1 \times 10^{-5}\). GRPO was executed with LoRA (rank 16, alpha 32) for 1 epoch, generating 8 rollouts per prompt with a per-device batch size of 8 and a learning rate of \(5 \times 10^{-6}\). The composite GRPO reward is formulated as: $\(R_{\text{total}} = 0.1 \cdot R_{\text{format}} + 0.2 \cdot R_{\text{global}} + 0.7 \cdot R_{\text{balanced\_rubric}}\)$ where \(R_{\text{balanced\_rubric}} = \frac{1}{2}(\text{Acc}_{\text{yes}} + \text{Acc}_{\text{no}})\), providing balanced gradients that discourage over-predicting affirmative tokens.
Key Experimental Results¶
Main Results¶
On the AnyAudio-Judge Bench, baseline LALMs and AnyAudio-Judge were evaluated across the symmetric Chinese (zh) and English (en) evaluation subsets using classification accuracy (ACC %). Both holistic prompting and dynamic rubric prompting were benchmarked.
| Model | Evaluation Mode | zh Avg | en Avg | SPEECH-REAL (zh/en) | SOUND-GEN (zh/en) | MUSIC-REAL (zh/en) | MIX (zh/en) |
|---|---|---|---|---|---|---|---|
| Audio-Flamingo3 | holistic | 57.97 | 59.61 | 50.25 / 48.87 | 67.66 / 67.58 | 59.97 / 69.32 | 52.11 / 51.65 |
| Audio-Flamingo3 | dynamic rubric | 63.91 | 64.19 | 60.25 / 59.00 | 68.92 / 68.42 | 70.28 / 70.97 | 56.50 / 57.20 |
| MiDashengLM | holistic | 50.75 | 49.71 | 51.58 / 48.58 | 49.50 / 51.33 | 52.22 / 50.56 | 47.20 / 47.60 |
| MiDashengLM | dynamic rubric | 68.35 | 67.94 | 66.94 / 64.92 | 75.83 / 74.75 | 71.25 / 72.78 | 59.90 / 59.90 |
| Kimi-Audio-7B-Instruct | holistic | 50.93 | 50.16 | 51.00 / 50.25 | 50.25 / 49.58 | 52.08 / 50.83 | 49.00 / 48.60 |
| Kimi-Audio-7B-Instruct | dynamic rubric | 70.84 | 70.81 | 81.17 / 80.08 | 75.17 / 74.00 | 69.72 / 70.42 | 64.90 / 65.80 |
| Qwen3-Omni-30B-Captioner | holistic | 65.33 | 64.24 | 75.42 / 70.46 | 64.00 / 66.58 | 68.61 / 67.92 | 55.30 / 51.90 |
| Qwen3-Omni-30B-Captioner | dynamic rubric | 76.66 | 76.77 | 90.17 / 88.50 | 76.42 / 78.42 | 74.86 / 78.33 | 69.10 / 66.70 |
| Gemini-2.5-Pro | holistic | 80.01 | 77.72 | 92.33 / 91.22 | 76.40 / 72.60 | 85.63 / 83.25 | 73.40 / 69.80 |
| Gemini-2.5-Pro | dynamic rubric | 78.31 | 77.27 | 90.67 / 89.00 | 77.00 / 76.00 | 79.67 / 79.33 | 75.50 / 71.00 |
| AnyAudio-Judge (Ours) | dynamic rubric | 85.26 | 84.45 | 93.33 / 91.42 | 82.50 / 84.67 | 92.22 / 91.25 | 90.60 / 86.40 |
Ablation Study and External Generalization¶
The ablation study isolates the contributions of the dynamic rubric prompting paradigm, supervised fine-tuning, and GRPO reinforcement learning. External human preference generalization is validated on the PAM dataset using Pearson (LCC), Spearman (SRCC), and Kendall's tau (KTAU).
| Evaluation & Training Configuration | Chinese ACC (%) | English ACC (%) | Performance Gain / Notes |
|---|---|---|---|
| Holistic judgment (Base Baseline) | 65.33 | 64.24 | Direct yes/no prompt on Qwen3-Omni-30B-Captioner |
| + Dynamic rubric (Paradigm Swap) | 76.66 | 76.77 | Zero-shot rubric decomposition and aggregation (+11.33 / +12.53) |
| + SFT (Supervised Fine-Tuning) | 84.02 | 83.78 | 105K corpus training with explicit rationales (+7.36 / +7.01) |
| + SFT + GRPO (Full Framework) | 85.26 | 84.45 | Policy optimization on 8,454 ambiguous hard negatives (+1.24 / +0.67) |
| Model / Metric (External PAM Dataset) | LCC โ | SRCC โ | KTAU โ |
|---|---|---|---|
| CLAPScore | 0.472 | 0.477 | 0.337 |
| AQAScore (Qwen2.5-Omni-7B) | 0.518 | 0.589 | 0.429 |
| AQAScore (AF3-Think) | 0.582 | 0.587 | 0.419 |
| AnyAudio-Judge (Ours) | 0.614 | 0.601 | 0.435 |
Key Findings¶
- Rubric decomposition delivers universal gains across open-source LALMs: Switching from holistic prompting to dynamic rubric evaluation yields dramatic accuracy improvements of 10% to 20% across virtually all open-source models (e.g., MiDashengLM jumps from ~50% to ~68%, Kimi-Audio jumps from ~50% to ~70%). This confirms that base LALMs often possess acoustic perceptual capacity but struggle to jointly evaluate composite constraints under monolithic prompts.
- Superior discrimination on complex mixed audio and generative outputs: In the cinematic MIX subset, where holistic models fail near chance levels (51%โ55%), AnyAudio-Judge achieves 90.60% (zh) and 86.40% (en). Similarly, on SOUND-GEN, accuracy surges to 82.50% / 84.67%, successfully exposing subtle generation defects such as missing secondary sounds and chronological scrambling.
- Effective dense reward for generative RL: Employing AnyAudio-Judge as a reward model to optimize DiTAR via GRPO improves its InstructTTSEval score from 69.6 to 72.9 (evaluated by Gemini) and achieves a win rate of 34.6% vs 26.9% against the base model in human preference evaluations.
Highlights & Insights¶
- From black-box scalars to an auditable rubric matrix: Rather than forcing a high-dimensional perception problem into a single opaque score, AnyAudio-Judge transforms evaluation into an explicit checklist, offering precise causal attribution by answering which exact attribute failed alongside grounding evidence.
- Soft satisfaction probabilities as dense RL rewards: Using the softmax difference of candidate logits provides continuous, fine-grained gradient feedback that avoids the optimization plateaus associated with discrete binary judgements.
- Rigorously mined hard negatives: The bench explicitly incorporates real-world synthesis failures and subtle, multi-axis attribute perturbations, preventing models from relying on superficial acoustic artifacts like background silence or clipping.
Limitations & Future Work¶
- Cascading dependency on rubric decomposition quality: The evaluation accuracy remains contingent on the upstream LLM's capacity to cleanly decouple captions. Incomplete decompositions or over-segmentations can compromise downstream audit validity.
- Elevated inference latency: Evaluating \(n\) individual rubric items sequentially or via concatenated contexts increases computational overhead compared to single-pass holistic inference.
- Future Directions: Developing unified, end-to-end architectures that perform simultaneous instruction parsing and multi-aspect cross-modal verification within a single forward pass.
Related Work & Insights¶
- vs CLAPScore / Global Embeddings: CLAP measures broad semantic proximity in a shared embedding space but remains insensitive to fine-grained compositional constraints; AnyAudio-Judge provides structured, verifiable attribution.
- vs AQAScore / InstructTTSEval: Prior benchmarks either focus strictly on single modalities (e.g., speech-only prosody or sound-only QA); AnyAudio-Judge provides a unified evaluation standard spanning speech, sound, music, and mixed audio.
- vs Text-based Rubric Evaluators: Unlike static or code-centric rubrics (e.g., AutoRubric, LLM-Rubric), AnyAudio-Judge successfully establishes dynamic, instance-specific rubric evaluation over complex multimodal audio signals.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering dynamic rubric decomposition and soft probability aggregation for comprehensive audio instruction alignment.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous bilingual 7,920-sample hard-negative benchmark, 105K training corpus, external PAM validation, and downstream RL ablation.
- Writing Quality: โญโญโญโญโญ Exceptionally structured presentation with clear mathematical definitions, thorough prompt appendices, and precise figures.
- Value: โญโญโญโญโญ Provides both a much-needed discriminative benchmark and an actionable, interpretable dense reward signal for the generative audio community.