Skip to content

Beyond Text Following: Repairable Arbitration Reversals in Audio-Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Audio-language models, multimodal evidence arbitration, repairable arbitration reversals, activation patching, training-free logit correction

TL;DR

This work demonstrates that text-following errors in audio-language models during multimodal conflict stem from arbitration readout reversals rather than perceptual failures, and introduces GACL, a training-free gated counterfactual decoding rule that rescues audio-supported answers under strict faithfulness budgets.

Background & Motivation

Audio-language models (ALMs) are expanding rapidly from conventional near-field speech transcription into high-stakes agentic settings such as automated meeting transcription assistance, emergency department triage, and multimodal robotics. In these interactive workflows, models frequently ingest acoustic waveforms alongside structured textual context, such as meeting agendas, historical inspection logs, or incident intake notes. However, recent evaluation benchmarks—most notably MCR-Bench and ALME—have uncovered a catastrophic vulnerability: when an accompanying textual claim contradicts unambiguous auditory evidence, state-of-the-art ALMs almost invariably favor the text claim. Even when an audible acoustic signal (such as a 23-second emergency siren) is explicitly present, models frequently repeat the conflicting text asserting that no alarm was triggered.

This pervasive text-following behavior suffers from fundamental diagnostic ambiguity when viewed purely through black-box evaluation. It remains unclear whether the model failed to encode the auditory signal during acoustic feature extraction and cross-modal projection, or whether the acoustic evidence was faithfully represented internally but subsequently suppressed by strong language-prior dominance during final answer arbitration. Distinguishing between these two failure modes is of paramount engineering importance: a perceptual failure demands architectural overhauls and massive acoustic pretraining, whereas an arbitration failure calls for principled decoding-time conflict resolution. Prior contrastive decoding methods (such as AAD and ACD in the audio domain, or VCD and OPERA in vision-language models) construct reference distributions by weakening, masking, or perturbing perceptual evidence. While these techniques penalize reliance on degraded features, they cannot address the counterfactual question: when acoustic evidence is held completely intact, how does the model's decision shift when only the conflicting text is removed?

To bridge this diagnostic divide, this paper investigates the same-audio counterfactual directly. By keeping acoustic inputs identical and stripping away solely the contradictory textual context across five open-weight omni-modal architectures and four conflict benchmarks, the authors discover that 64.1% of conflict samples undergo a clear sign flip: the model vigorously endorses the audio-supported candidate in isolation, yet completely succumbs to the text-supported option when the joint text context is appended. Core idea: text-following errors are predominantly repairable arbitration reversals where acoustic evidence is preserved internally but suppressed at the decision boundary, enabling a training-free gated counterfactual decoding rule (GACL) that bridges answer-position residual steering to candidate-score differences for robust inference-time recovery.

Method

Overall Architecture

GACL operates via a two-branch causal counterfactual formulation over frozen model weights: a joint conflict branch \(\text{Joint (J)}\) conditioned on the audio, task query, and conflicting text description; and an audio-reference branch \(\text{Audio-Ref (A)}\) preserving the exact acoustic waveform, task query, and candidate set while removing solely the adversarial text. The framework leverages activation patching insights to establish that internal hidden corrections project directly onto observable candidate score differences, allowing dual gates (branch disagreement and reference reliability) to dynamically govern bounded convex logit interpolation at generation time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input<br/>Audio x + Text Description + Question q"] --> B["Two-Branch Counterfactual Scoring<br/>Frozen ALM yields Joint sJ and Audio-Ref sA Logits"]
    B --> C["Residual-Stream Causal Localization<br/>Verify Hidden Repair Direction Aligns with sA - sJ"]
    C --> D["Dual-Gated Convex Correction GACL<br/>Disagreement Gate Nout and Reliability Gate RA"]
    D -->|Bounded Interpolation Update| E["Corrected Model Output<br/>Recover Audio-Supported Answer ya"]

Key Designs

1. Two-Branch Diagnostic and Signed Margin Reversals: Disentangling Perceptual Failure from Arbitration Loss

To mathematically formalize behavioral arbitration failure, the framework evaluates two distinct forward passes sharing identical audio \(x\), question \(q\), and candidate set \(\mathcal{C}\). For candidates \((y_a, y_t)\) denoting the audio-supported and text-supported verbalizers respectively, signed margins are computed as \(M_A = s_A(y_a) - s_A(y_t)\) and \(M_J = s_J(y_a) - s_J(y_t)\) using length-normalized log-probabilities. The diagnostic operational space is formulated as:

\[\mathcal{O} = \{x \mid M_A > 0, M_J < 0\}\]

Within \(\mathcal{O}\), \(M_A > 0\) demonstrates that the acoustic encoder independently captures the requisite evidence to favor \(y_a\), while \(M_J < 0\) reveals that the presence of conflicting text overrides the auditory evidence. Across five frontier models, 64.1% of all conflict instances fall into this quadrant, proving that text following is largely a reversible decision-making arbitration failure rather than an acoustic blindness defect.

2. Answer-Boundary Residual Localization and Rank Alignment: Bridging Hidden States to Output Scores

To identify the precise internal locus of arbitration, activation patching is applied across three token functional sites: audio-span tokens, prompt instruction tokens, and the answer-position residual stream immediately preceding generation. Injecting reference hidden states at the answer boundary achieves an 81% macro rescue rate across models, whereas intervening on audio-span and instruction tokens yields exactly 0.00% rescue. Probing audits further demonstrate that \(y_a\) remains linear-decodable throughout late layers, but cross-branch decoding collapses during a mid-to-late "commit window," indicating that readout orientation is dynamically rewritten. Crucially, the internal patch displacement vector correlates almost perfectly with the observable output candidate difference \(s_A - s_J\) at rank level (Spearman \(\rho = 0.93\)), demonstrating that output logit differences serve as a faithful, non-invasive surrogate for the internal mechanistic repair direction.

3. Gated Audio Counterfactual Logit Correction (GACL): Robust Inference-Time Convex Steering

To convert the diagnostic displacement into a practical decoding algorithm that safeguards against hallucination on consistent inputs, GACL introduces a dual-gating mechanism with convex boundedness. The sequence-level branch-disagreement gate \(N_{\text{out}}(x) = \mathbb{I}[\hat{y}_A \neq \emptyset] \cdot \mathbb{I}[\hat{y}_A \neq \hat{y}_J]\) silences intervention when both branches already agree. The reference-reliability gate \(R_A(x) = \text{clip}(\Delta_A(x) / \tau_A, 0, 1)\) scales correction by the margin \(\Delta_A(x) = s_A(\hat{y}_A) - \max_{y \neq \hat{y}_A} s_A(y)\), automatically attenuating intervention whenever the acoustic reference exhibits low confidence. The unified interpolation weight is:

\[\alpha(x) = \text{clip}(\lambda \cdot R_A(x) \cdot N_{\text{out}}(x), 0, 1)\]

At generation step \(k\), the corrected logits are computed as \(z_{k}^{\text{GACL}} = z_{J,k} + \alpha(x) (z_{A,k} - z_{J,k})\). Clamping \(\alpha \in [0, 1]\) prevents out-of-manifold extrapolation and syntax collapse, ensuring strict adherence to the output manifold.

Benchmark & Evaluation Protocol

Evaluation spans five open-weight multimodal ALMs: Qwen2-Audio-7B-Instruct, Qwen2.5-Omni-7B, Voxtral-Small-24B, Qwen3-Omni-30B-A3B-Instruct, and Kimi-Audio-7B-Instruct. Tasks include AQA (Audio Question Answering), VSC (Vocal Sound Classification), SER (Speech Emotion Recognition) from MCR-Bench, alongside speech-transcript conflict from ALME-English. Performance is quantified along a rescue–faithfulness Pareto frontier using normalized area under the curve \(\text{nAUC}(K_{\max}) = \frac{1}{K_{\max}} \int_0^{K_{\max}} R(K) dK\), where \(R(K)\) represents peak conflict accuracy gain subject to a faithful drop constraint \(\Delta_{\text{faith}} \le K\text{ pp}\). The strict threshold is set to \(K \le 5\text{ pp}\) and relaxed to \(K \le 10\text{ pp}\).

Key Experimental Results

Main Results

GACL is compared against the uncorrected Joint baseline and state-of-the-art contrastive decoding baselines AAD (omitting audio) and ACD (perturbing audio). Across 40 evaluation settings, GACL attains top performance in 39 combinations, delivering an average improvement of 17.8 nAUC points over the strongest contrastive baseline under the strict 5 pp budget:

Model Method AQA (0-5 / 0-10) VSC (0-5 / 0-10) SER (0-5 / 0-10) ALME (0-5 / 0-10) Average (0-5 / 0-10)
Qwen2-Audio-Inst AAD 12.1 / 18.9 18.4 / 29.4 2.3 / 3.3 39.5 / 43.7 18.0 / 23.9
ACD 8.1 / 9.1 5.6 / 10.4 0.0 / 0.0 21.5 / 26.8 8.8 / 11.6
GACL 51.0 / 60.7 39.1 / 51.2 2.9 / 6.1 60.4 / 63.4 38.3 / 45.4
Qwen2.5-Omni AAD 14.8 / 28.1 48.4 / 56.6 3.4 / 6.8 54.8 / 56.4 30.4 / 37.0
ACD 0.5 / 0.5 29.4 / 37.0 3.9 / 5.7 32.8 / 41.2 16.7 / 21.1
GACL 31.6 / 45.7 58.0 / 71.6 7.6 / 11.4 66.0 / 66.0 40.8 / 48.7
Voxtral-Small AAD 0.0 / 2.8 7.1 / 8.5 2.8 / 4.4 17.9 / 29.4 7.0 / 11.3
ACD 1.7 / 3.1 0.5 / 0.5 2.3 / 2.9 2.1 / 7.0 1.6 / 3.4
GACL 8.8 / 17.4 25.0 / 28.1 6.7 / 10.3 72.0 / 76.0 28.1 / 33.0
Qwen3-Omni AAD 11.8 / 20.1 7.0 / 14.8 0.0 / 0.0 44.7 / 45.0 15.9 / 20.0
ACD 8.1 / 8.2 0.7 / 0.9 0.8 / 1.2 5.2 / 5.2 3.7 / 3.9
GACL 30.1 / 38.6 36.2 / 42.5 6.6 / 10.2 64.0 / 64.0 34.2 / 38.8
Kimi-Audio-Inst AAD 10.5 / 17.2 12.3 / 24.8 0.5 / 0.7 2.0 / 4.2 6.3 / 11.8
ACD 5.4 / 8.6 1.7 / 3.2 0.8 / 1.1 2.2 / 4.1 2.5 / 4.2
GACL 39.3 / 48.7 12.7 / 19.2 2.0 / 4.9 47.0 / 47.0 25.2 / 30.0

Ablation & Stress Analysis

Ablation analysis on the \(5 \times 4\) model-task grid isolates the individual failure surfaces guarded by each algorithmic component under tight operational budgets:

Variant Faithful Drop @ 5 pp (pp) ↓ Faithful Drop @ 2.5 pp (pp) ↓ Low-\(R_A\) Drop (pp) ↓ Free-Form Preserve (%) ↑ SER Parse Fail (%) ↓ Average Gain (pp) ↑
Joint (no correction) 0.0 0.0 0.0 100.0 0.0 0.0
Hard switch (endpoint) 13.3 Infeasible — — — 66.5
Naive interp. (fixed \(\alpha\)) 4.6 3.9 4.7 70.0 7.0 38.4
w/o bound (\(\alpha \le \infty\)) 2.1 1.8 2.1 90.0 27.0 46.4
w/o \(N_{\text{out}}\) (disagreement gate) 2.1 1.8 2.1 22.0 0.0 46.1
w/o \(R_A\) (reliability gate) 2.3 2.3 3.2 90.0 0.0 47.4
GACL (full) 2.1 1.8 2.1 90.0 0.0 45.8

Key Findings

  • Component-Specific Failure Prevention: Omission of the reliability gate \(R_A\) elevates faithful accuracy loss on weak-audio inputs from 2.1 pp to 3.2 pp, breaching the strict 2.5 pp budget constraint. Discarding the disagreement gate \(N_{\text{out}}\) degrades free-form exact surface preservation from 90.0% down to 22.0% due to gratuitous alterations on concordant predictions. Removing the upper bound (\(\alpha > 1\)) triggers a 27.0% parsing failure rate on SER due to out-of-distribution logit extrapolation.
  • Competitiveness with Supervised Fine-Tuning: Against a domain-adapted LoRA baseline on Qwen2-Audio-7B (which achieves 65.3% adversarial accuracy at an 11.0 pp faithful degradation cost), training-free GACL retains 76% of LoRA's adversarial improvement (reaching 54.0% accuracy) while strictly containing faithful degradation.
  • Zero-Shot Cross-Modal Transfer to Vision-Text Conflict: Applying GACL with invariant default parameters \((\lambda=0.5, \tau_A=0.5)\) to visual question answering conflict on MC2 boosts adversarial accuracy by +40.5 pp on Qwen3-VL-2B and +26.5 pp on Qwen3-VL-8B at essentially zero faithful cost (\(\le 0.5\text{ pp}\)).

Highlights & Insights

  • Counterfactual Diagnosis Overturning Perceptual Dogma: Proves conclusively that multimodal text bias is largely rooted in late decision-boundary arbitration overrides rather than low-level acoustic feature degradation.
  • Direct Bridge from Residual Patching to Output Logits: Uncovers an empirical rank alignment (\(\rho = 0.93\)) between internal residual steering vectors and observable output score deltas, bypassing internal hooks at inference time.
  • Conservative Gating Enabling Safe Test-Time Deployment: Demonstrates that pairing a disagreement detector with reference confidence gating permits aggressive conflict recovery without catastrophic degradation on consistent inputs.

Limitations & Future Work

  • Requirement for Clean Text Isolation: The same-audio counterfactual presupposes that conflicting textual premises can be identified and excised from the prompt, which may be challenging in uncurated web retrieval contexts.
  • Dual-Branch Forward Latency Overhead: Running two inference branches incurs an extra forward pass; though caching shared audio-encoder states curtails latency overhead to \(1.27\times\), single-pass compilation remains an open direction.
  • Inability to Mend Inherent Acoustic Blind Spots: GACL operates as an arbitration corrector and cannot synthesize auditory comprehension when the underlying acoustic encoder fails entirely.
  • vs Audio Contrastive Decoding (AAD / ACD): Traditional audio contrastive methods subtract no-audio or perturbed-audio distributions to mitigate hallucinations, which is misaligned for arbitration conflict; GACL isolates the text bias by using a same-audio reference.
  • vs Vision-Language Contrastive Decoding (VCD / OPERA): Prior multimodal decoding schemes perturb sensory modalities; GACL flips this paradigm by perturbing the linguistic context to expose latent multimodal competence.
  • vs Parameter-Efficient Fine-Tuning (LoRA): Supervised adaptation risks catastrophic forgetting on general conversational benchmarks and requires paired conflict corpora; GACL provides a zero-training, plug-and-play alternative.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates repairable arbitration reversals and connects mechanistic activation patching to output logit correction.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thoroughly executed across 5 model architectures, 4 audio tasks, mechanistic ablation grids, and cross-modal vision-text transfer.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative arc, moving seamlessly from empirical diagnosis to mechanistic localization and practical algorithm design.
  • Value: ⭐⭐⭐⭐⭐ Establishes a foundational paradigm for multimodal conflict arbitration and safe, budget-constrained test-time calibration.