Skip to content

Overtone: Fine-Grained Audio Reasoning via Diagnostic-Driven Delayed Acoustic Fusion

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Audio & Speech
Keywords: Audio Large Language Models, Representation Reorganization, Delayed Acoustic Fusion, Gated Cross-Attention, Fine-Grained Audio Reasoning

TL;DR

Addressing the "representation reorganization" bottleneck in audio LLMs—where middle layers systematically overwrite fine-grained acoustics into semantic abstractions—Overtone introduces a diagnostic-driven delayed acoustic fusion architecture that extracts dual semantic/acoustic tracks from model-localized segments and injects frozen WavLM features into post-compression layers (L12–L24) via gated cross-attention with reconstruction regularization, achieving state-of-the-art results on MMAU among open-source \(\le\)8B models (77.66%).

Background & Motivation

Audio is inherently among the richest communication modalities, capturing not only symbolic semantic messages but also continuous paralinguistic and acoustic signals such as speaker vocal timbre, subtle pitch inflections, emotive nuance, and acoustic room acoustics. Recent end-to-end audio large language models (Audio LLMs)—exemplified by Qwen2.5-Omni, Audio-Flamingo, and Gemini—have successfully unified speech, general sound events, and music processing within autoregressive decoder architectures. However, an enduring asymmetry continues to limit these models: while they excel at semantic tasks such as automatic speech recognition and macro-level intent classification, their performance consistently degrades on tasks requiring fine-grained acoustic discrimination, including speaker identity matching, pitch contour tracking, and temporal acoustic rhythm parsing. Scaling up parameter counts or training on more diverse audio mixtures fails to eliminate this performance ceiling.

To understand the structural origin of this bottleneck, the authors conducted a rigorous layer-wise linear probing study across representative audio LLMs (including Qwen2.5-Omni 7B, Audio-Flamingo-3 8B, Qwen3-Omni 30B, and Step-Audio-R1 33B). The findings reveal a pervasive internal phenomenon termed "representation reorganization": within the model's shallow-to-middle layers (specifically between Layer 6 and Layer 12 in 28-layer architectures), semantic discriminability (measured by Keyword Content probe accuracy) surges from 22.8% at the encoder interface to 64.4% and stabilizes, while fine-grained acoustic attributes (Speaker ID and Pitch Level) simultaneously suffer an abrupt collapse of nearly 50 percentage points. Driven by the language modeling objective of next-token text prediction, the network systematically reallocates its representational capacity toward semantic abstractions, overwriting and purging low-level acoustic details. Recent re-listening paradigms and reinforcement learning deliberation approaches (such as Echo and Audio-Thinker) re-inject audio into early layers, meaning the re-listened audio still traverses the middle-layer compression zone and forfeits the very acoustic cues needed for fine-grained discrimination.

Overcoming this fundamental bottleneck requires an architectural mechanism that directly bypasses the compression zone rather than relying on redundant data or unconstrained full-depth re-listening. The core idea is to diagnose the layer-wise compression boundary and implement a diagnostic-driven delayed acoustic fusion architecture (Overtone) that extracts dual-track representations from autonomously localized audio intervals, routing frozen self-supervised WavLM acoustic features directly into post-compression layers (L12, 16, 20, 24) via gated cross-attention, anchored by deep reconstruction regularization and refined through policy optimization.

Method

Overall Architecture

The Overtone pipeline operates on demand in three structured phases: First-Pass CoT and Re-listening Triggering, Dual-Track Feature Extraction, and Delayed Acoustic Fusion with Second-Pass Generation. Given an input audio waveform and a user query, the model generates an initial chain-of-thought (CoT). When acoustic verification is necessary, the model emits a structured temporal tag <seg>s,e</seg> delineating the critical interval \([s, e]\). The identified segment is extracted from the raw waveform and processed through two independent paths: a semantic track (\(X_{sem}\)) via the LLM's standard audio encoder and an acoustic track (\(X_{aco}\)) via a frozen WavLM-Large encoder. In the second forward pass, \(X_{sem}\) enters from Layer 0 as standard audio tokens, while \(X_{aco}\) is cached in an acoustic memory buffer and queried via gated cross-attention specifically at layers 12, 16, 20, and 24, allowing the model to formulate its final judgment grounded in high-fidelity acoustic evidence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    IN["Raw Audio Waveform + Question"] --> P1["First Pass: CoT Generation"]
    P1 --> D1["Learned Re-listening Trigger<br/>RL-refined segment tag [s, e]"]
    D1 --> DUAL["Dual-Track Feature Extraction<br/>Semantic Qwen + Acoustic WavLM"]
    DUAL --> D2["Gated Delayed Acoustic Fusion<br/>Post-compression layers L12-L24 GQA"]
    D2 --> D3["Deep Reconstruction Regularization<br/>Final hidden state cosine alignment"]
    D2 --> OUT["Second Pass: Verifiable Decision"]
    D3 -.->|Gradient constraint on top layers| D2

Key Designs

1. Learned Re-listening Trigger: Autonomous end-to-end temporal segment localization replacing heuristic and whole-clip re-listening
Heuristic re-listening or reprocessing full-length audio introduces substantial latency overhead and dilutes discriminative acoustic details within global background noise. Overtone delegates the decision of "when" and "where" to re-listen directly to the LLM itself. During initial CoT generation, the model predicts temporal boundaries \([s, e]\) via <seg>s,e</seg>. To resolve the absence of ground-truth reasoning trajectories and prevent hallucinated timestamps, localization is trained in two decoupled stages: first, an SFT cold-start on synthetic QA pairs verified through cross-model consensus and silent-audio filtering (discarding 51% of candidate samples answerable without audio priors); second, Group Relative Policy Optimization (GRPO) using a composite reward: $\(R = R_{acc} + R_{format} + R_{seg} + R_{penalty}\)$ Here \(R_{acc}\) enforces answer accuracy, \(R_{format}\) rewards valid tag syntax, \(R_{seg} = w_{seg} \cdot \text{IoU}(\text{pred}, \text{oracle})\) directly optimizes boundary overlap, and \(R_{penalty}\) suppresses out-of-bounds or malformed outputs. This end-to-end policy optimization yields a mean segment IoU of 0.93 with oracle intervals. At inference time, a lightweight 3-layer MLP Restart Controller evaluates first-pass uncertainty and suppresses unnecessary second passes, reducing computational overhead while retaining precision.

2. Gated Delayed Acoustic Fusion: Bypassing the core compression zone via sparse post-compression gated cross-attention
Guided by the layer-wise probing diagnosis identifying layers 6–12 as the core compression zone, Overtone avoids early-stage acoustic concatenation. For the identified segment \([s, e]\), the acoustic track utilizes a frozen WavLM-Large backbone to generate raw acoustic embeddings \(Z_{aco} \in \mathbb{R}^{T \times 1024}\), projected into LLM hidden dimension via linear transformation and LayerNorm: \(X_{aco} = \text{LN}(W_{proj} Z_{aco} + b_{proj}) \in \mathbb{R}^{T \times 3584}\). At four designated post-compression layers \(l \in \{12, 16, 20, 24\}\), Overtone appends a gated grouped-query cross-attention (GQA, 28 query heads, 4 KV heads) after standard self-attention: $\(\hat{H}_{l,t} = H'_{l,t} + g_{l,t} \cdot \text{CrossAttn}_l(Q=H'_{l,t}, K=X_{aco}, V=X_{aco})\)$ where the token-specific routing scalar is defined as \(g_{l,t} = \sigma(W_l \text{RMSNorm}(H'_{l,t}) + b_l)\). Crucially, the parameters are initialized with \(W_l = 0\) and \(b_l = -5\), resulting in \(g_{l,t} \approx 0.007\) at the start of training. This initialization guarantees that the base model's pretrained semantic and reasoning faculties remain undisturbed, progressively opening the gate only for tokens that actively demand fine-grained acoustic evidence.

3. Deep Reconstruction Regularization: Eliminating semantic shortcuts by constraining deep hidden states to reconstruct acoustic targets
In complex multi-task audio-language modeling, task-level cross-entropy loss \(L_{QA}\) provides only an indirect and sparse supervision signal for the bypass channel. Consequently, the model risks learning semantic shortcuts—guessing answers from linguistic context or high-level semantics while ignoring the injected acoustic representations. To enforce genuine utilization of the bypass, Overtone introduces an explicit representation reconstruction objective during SFT: $\(\mathcal{L}_{recon} = -\frac{1}{T} \sum_{t=1}^{T} \cos\left(f_\theta(H_L^{(t)}), \text{sg}(Z_{aco}^{(t)})\right)\)$ where \(H_L^{(t)}\) represents the final-layer hidden state (output of the final RMSNorm) at audio token positions, \(f_\theta\) is a 2-layer MLP projection head (\(3584 \to 2048 \to 1024\) with GELU activation), and \(\text{sg}(\cdot)\) is the stop-gradient operator applied to raw WavLM features. This loss penalizes representations that lose predictive capacity over raw acoustic features after 28 Transformer layers. In ablation studies, pairing delayed fusion with \(\mathcal{L}_{recon}\) yields a +4.7 pp improvement, whereas imposing reconstruction on early-fusion architectures creates optimization interference and degrades performance.

A Worked Example

Consider Case 2 from the paper (Pitch Comparison on continuous speech):
The input audio contains two consecutive speech segments spanning \(0.0\) to \(7.3\) seconds. The query asks: "How does the pitch of the second speech segment compare to the first? (A: Same pitch; B: Higher pitch; C: Cannot determine; D: Lower pitch; Ground truth: D)".
1. First Pass: In the first forward pass, Qwen2.5-Omni operates on representations compressed through middle layers; it mistakes the fundamental frequency contour and outputs an erroneous initial answer B, while generating the tag <seg>0.00,7.30</seg>.
2. Controller Routing & Dual Extraction: The Restart Controller detects high answer uncertainty, approving the second pass. The \([0.0, 7.3]\text{ s}\) waveform segment is routed to frozen WavLM-Large, extracting continuous uncompressed F0 and harmonic features into acoustic memory buffer \(X_{aco}\).
3. Delayed Injection: During the second pass, as the decoder reaches layers 12, 16, 20, and 24, the token-dependent gates open for pitch-evaluating tokens, querying \(X_{aco}\) via cross-attention.
4. Corrected Generation: Supplied with uncompressed pitch contours, the second-pass CoT identifies the descending pitch contour and outputs the correct answer D.

Loss & Training

Overtone employs a two-stage training strategy: - Stage 1 (SFT Cold Start): Trains the model on 75k audio-grounded QA pairs using \(\mathcal{L}_{SFT} = \mathcal{L}_{QA} + \lambda \cdot \mathcal{L}_{recon}\) with \(\lambda = 0.1\). The LLM backbone is fine-tuned via LoRA (rank 16, \(\alpha = 32\)) on attention and MLP projections, while the gated cross-attention modules and projection layer are trained from scratch. Training runs for 5 epochs on 8 \(\times\) A100-80GB GPUs with learning rate \(5 \times 10^{-6}\), effective batch size 64, and bf16 precision. - Stage 2 (GRPO Policy Refinement): Focuses on autonomous localization refinement with \(\lambda = 0\). The policy generates 8 rollouts per prompt with a KL divergence penalty (\(\beta = 0.04\)) and temperature 1.0, optimizing standardized group advantage \(A_i = (R_i - \mu_R)/\sigma_R\).

Key Experimental Results

Main Results

The table below presents comprehensive evaluation results on MMAU (spanning Speech, Sound, and Music domains) and MMAR (cross-domain reasoning). Overtone achieves the highest overall accuracy on MMAU among open-source \(\le\)8B models, outperforming the 30B Qwen3-Omni and competing directly with leading proprietary systems.

Model Size MMAU Speech (%) MMAU Sound (%) MMAU Music (%) MMAU Avg (%) MMAR Avg (%)
Proprietary Flagships
GPT-4o-Audio – 69.33 63.20 49.93 60.82 64.09
Gemini-2.5-Pro – 72.67 70.63 64.77 69.36 70.03
Step-Audio-2 – 72.75 80.60 68.23 73.86 –
Open-source (Large Scale)
Qwen3-Omni 30B 70.70 77.87 66.83 71.80 65.50
Step-Audio-R1 33B 76.50 85.24 71.27 77.67 69.15
Open-source (\(\le\)8B)
Qwen2.5-Omni (Base) 7B 68.90 76.77 67.33 71.00 57.33
Audio-Flamingo-3 8.2B 66.97 75.83 74.47 72.42 58.50
AF-Next-Captioner 8B 72.13 79.87 75.30 75.76 63.00
MiMo-Audio 7B 68.47 82.58 73.65 74.90 63.60
Omni-R1 7B 75.80 78.30 70.80 74.97 63.46
Audio-Thinker 7B 76.60 79.03 70.53 75.39 67.25
Echo 7B 77.87 79.62 72.33 76.61 69.99
Overtone (Ours) 7.4B 80.97 77.87 74.13 77.66 69.58

On supplementary benchmarks, Overtone maintains its marked advantage in fine-grained perception: on MMAU-Pro, accuracy improves by +7.6 pp overall (with a +7.5 pp gain on open-ended QA); on the fine-grained spoken language benchmark MMSU, Phonology Perception leaps by +30.4 pp (37.6% \(\to\) 68.0%) and Paralinguistics Perception increases by +26.9 pp (39.5% \(\to\) 66.4%).

Ablation Study

Controlled ablation experiments on MMAU-mini isolate each architectural component, reporting accuracy and \(\Delta\) relative to the 81.5% SFT full-model baseline:

Group Configuration Acc. (%) \(\Delta\) (pp) Note
(a) Fusion Strategy Base model (zero-shot) 56.5 −25.0 Pretrained baseline without fine-tuning
SFT-only (no acoustic bypass) 70.3 −11.2 Standard LoRA without bypass channel
Early fusion, no \(\mathcal{L}_{recon}\) 75.0 −6.5 Input-level injection at Layer 0
Early fusion + \(\mathcal{L}_{recon}\) 72.4 −9.1 Optimization conflict in compression zone
Delayed fusion, no \(\mathcal{L}_{recon}\) 76.8 −4.7 Bypass at L12 without feature constraint
Delayed fusion + \(\mathcal{L}_{recon}\) 81.5 0.0 Full SFT baseline configuration
(b) Re-listening Strategy Raw re-listen (standard encoder) 74.4 −7.1 Re-injecting audio through full LLM depth
Oracle segmentation WavLM bypass 82.3 +0.8 Upper bound with ground-truth intervals
Learned WavLM bypass (Overtone) 81.5 0.0 Autonomous localization via <seg> tags
(c) Injection Layer Start Early fusion (L0 Start) 75.0 −6.5 Exposed directly to reorganization zone
L4 Start (4, 8, 12, 16) 73.8 −7.7 Early compression zone entrance
L8 Start (8, 12, 16, 20) 79.9 −1.6 Mid-compression zone
L12 Start (12, 16, 20, 24) 81.5 0.0 Optimal post-compression placement
L16 Start (16, 20, 24, 27) 76.7 −4.8 Insufficient remaining Transformer depth
L20 Start (20, 22, 24, 26) 73.2 −8.3 Very deep injection limits cross-attention
(d) Acoustic Causality Normal bypass (unperturbed) 78.7 −2.8 Single-pass evaluation baseline
Shuffled acoustic features 71.1 −10.4 Temporal structure randomized
Mean-pooled bypass 75.5 −6.0 Temporal dynamics collapsed
No acoustic input 68.9 −12.6 Bypass modules present but unconditioned
(e) Frozen Encoder Choice Whisper-Large-v3 (ASR supervised) 75.6 −5.9 Text-biased semantic speech features
HuBERT-Large (SSL acoustic) 78.3 −3.2 Masked prediction preserves acoustic details
WavLM-Large (Full-stack SSL) 81.5 0.0 Broadest fine-grained acoustic retention

Key Findings

  • Causal Acoustic Verification: Randomly shuffling acoustic features along the time axis causes a 10.4 pp accuracy drop (78.7% \(\to\) 71.1%), and completely withholding acoustic input drops accuracy to 68.9%. This confirms that Overtone's gains stem directly from genuine acoustic information flow rather than parameter capacity expansion.
  • Criticality of Injection Depth: Probing L12 confirms it as the optimal injection onset. Earlier injection exposes features to the compression zone, whereas injecting too late (L16 or L20) deprives the model of adequate intermediate reasoning layers to synthesize acoustic cues.
  • Self-Supervised vs. Supervised Representations: Replacing WavLM with ASR-supervised Whisper-Large-v3 causes a significant drop (−5.9 pp), proving that speech models optimized purely for linguistic transcription suffer from internal semantic compression and fail to provide the required acoustic foundation.
  • Latency and Trigger Selectivity: Overtone adds only +0.36 s of average inference latency (~18% increase over SFT-only). The model selectively invokes re-listening based on acoustic complexity—triggering on 73.4% of speech tasks, 56.7% of music tasks, and 51.9% of sound tasks.

Highlights & Insights

  • Diagnostic-Guided Architecture Formulation: Rather than empirically stacking more encoders at the input stage, the authors first identified the "L6–L12 representation reorganization" compression zone via probing, designing an architectural bypass that directly responds to empirical evidence.
  • Zero-Initialized Gates with Reconstruction Anchor: Combining \(b=-5\) gate initialization with deep reconstruction regularization ensures seamless preservation of base LLM reasoning capabilities while forcing the network to maintain uncorrupted acoustic memory.
  • Autonomous Sub-Clip Localization via GRPO: Leveraging policy optimization with segment IoU rewards enables the model to locate and re-listen to critical audio intervals with 0.93 mean IoU, avoiding the severe computational burden of full-audio re-listening.

Limitations & Future Work

  • Local Over-Interpretation on Global Temporal Queries: As highlighted in failure analysis (Case 4), segment-level re-listening can cause the model to mistake local rhythmic fluctuations for global tempo trends, showing diminishing returns on holistic long-context reasoning.
  • Ineffectiveness on External Knowledge Tasks: The acoustic bypass cannot compensate for missing factual knowledge; tasks reliant on world facts (e.g., Eco-Acoustic Knowledge) show flat or slightly negative trends (−2.1 pp).
  • Gating Interpretability: Gates exhibit elevated activation even on seemingly semantic tasks such as event counting, indicating that temporal localization interactions are more complex than a binary semantic-versus-acoustic dichotomy.
  • vs. Echo (Wu et al., 2026a): While Echo introduces an audio-interleaved reasoning paradigm, it re-injects audio into the input layer, allowing middle layers to compress acoustic details anew; Overtone diagnoses this structural flaw and bypasses the compression zone directly.
  • vs. Audio-Thinker (Wu et al., 2026b): Audio-Thinker applies reinforcement learning to optimize reasoning compute but operates on representations already stripped of acoustic fidelity; Overtone establishes a preserved acoustic foundation prior to policy refinement.
  • vs. SALMONN (Tang et al., 2024): SALMONN fuses dual encoders at the input layer, where fine-grained speech cues are prematurely abstracted; Overtone demonstrates that semantic and acoustic tracks must interface at different model depths.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the first diagnostic-driven bypass architecture specifically targeted at internal representation reorganization in audio LLMs.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Includes extensive probing across four model scales, multi-benchmark evaluations, rigorous ablations, and insightful failure mode analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Outstanding clarity and cohesion, establishing an airtight narrative from diagnostic probing to architectural execution.
  • Value: ⭐⭐⭐⭐⭐ Provides a foundational blueprint for resolving the tension between deep semantic abstraction and fine-grained perception in multimodal foundation models.