Skip to content

Beyond Text Following: Repairable Arbitration Reversals in Audio-Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Audio & Speech
Keywords: Audio-Language Models, Multimodal Conflicts, Evidence Arbitration, Activation Patching, Training-free Decoding

TL;DR

Addressing the arbitration failure where audio-language models blindly follow conflicting text over clear acoustic evidence, this paper proposes Gated Audio Counterfactual Logit Correction (GACL), a training-free decoding rule grounded in causal activation patching that interpolates joint logits toward a same-audio reference branch to rescue audio decisions under strict faithfulness constraints.

Background & Motivation

As audio-language models (ALMs) advance beyond conventional speech transcription into high-stakes agentic workflows such as meeting assistance, emergency dispatch, and incident triage, they increasingly process acoustic signals paired with external textual collateral like agendas, incident logs, or operator notes. However, recent benchmark evaluations on suites such as MCR-Bench and ALME reveal a glaring behavioral vulnerability: whenever the textual description contradicts clear auditory signals (e.g., an alarm sounding in audio while the text asserts no alarm occurred), state-of-the-art ALMs exhibit an overwhelming propensity toward text-following, uncritically confirming the erroneous textual assertion. This observation raises a fundamental mechanistic dilemma: does this failure stem from an acoustic perceptual deficit where the model completely misses the auditory event, or does the model adequately encode the auditory evidence only to have it overridden during multimodal decision arbitration?

The core tension lies in the fact that perceptual degradation and arbitration failure demand diametrically opposed solutions. An acoustic perceptual failure necessitates upgrading speech encoders, pretraining objectives, or cross-attention capacity; conversely, an arbitration failure calls for targeted calibration at the decision boundary. Existing test-time correction schemesโ€”including audio-specific methods like AAD and ACD as well as visual contrastive counterparts like VCD and OPERAโ€”construct negative counterfactual references by artificially corrupting, masking, or eliminating the non-text modality. While effective for penalizing reliance on degraded sensory priors, these methods are conceptually incapable of testing multimodal conflict arbitration: they fail to evaluate what preference the model would exhibit if the acoustic evidence were held intact while only the competing textual claim was purged.

To isolate this arbitration mechanism, the authors formulate a same-audio counterfactual diagnostic: maintaining identical audio and question inputs, they eliminate solely the conflicting textual assertion and trace the resulting preference shifts. Across five representative open-weight ALMs and four conflict tasks, 64.1% of conflicting samples exhibit an unmistakable sign flipโ€”the uncorrupted audio-only reference branch reliably favors the audio-grounded answer, whereas the joint branch succumbs to the text. Causal activation patching further pinpoints this reversal to the residual stream at the final answer commitment position, demonstrating that the internal patch-induced repair direction aligns tightly with observable candidate score differences (Spearman rank correlation \(\rho = 0.93\)). The core idea is: leverage a frozen same-audio counterfactual branch as a parameter-free diagnostic, and employ Gated Audio Counterfactual Logit Correction (GACL)โ€”governed by branch disagreement and reference reliability gatesโ€”to interpolate joint predictions toward same-audio logits along the causal repair vector within safe convex bounds.

Method

To counteract arbitration reversal systematically, the authors translate mechanistic diagnostic insights into an inference-time decoding procedure. The overall framework takes a multimodal query containing raw audio, an adversarial text description, a query prompt, and a closed candidate label set, and processes it through a dual-branch structure followed by dual-gate validation and bounded logit interpolation.

Overall Architecture

During inference, the model executes two forward branches: the joint branch \(J\) conditions on the complete context (audio, conflicting text, and question), whereas the same-audio reference branch \(A\) (REF) drops the conflicting text while preserving the identical audio segment and task question. If both branches yield identical top-1 candidate predictions, the joint prediction is emitted directly without modification. However, when the branches diverge, the system activates a branch disagreement gate and measures the margin of the reference branch via a reference reliability gate. If the reference demonstrates sufficient acoustic confidence, GACL smoothly pulls the joint logits toward the reference logits using an adaptive, bounded interpolation weight \(\alpha(x)\), rescuing the audio-supported hypothesis without disturbing faithful inputs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Multimodal Input: Audio x + Text Context + Question q"] --> Branches["Dual-Branch Forward: Joint Branch J (A+T+Q) & Same-Audio Ref A (A+Q)"]
    Branches --> GateNout["1. Branch Disagreement Gate: Verify top-1 prediction divergence"]
    GateNout -->|Disagreement| GateRA["2. Reference Reliability Gate: Measure reference margin"]
    GateNout -->|Agreement| Direct["Retain unmodified Joint branch logits"]
    GateRA --> Interp["3. Bounded Convex Logit Interpolation: Safe logit steering toward reference"]
    Interp --> Out["Final Decoded Output: Rescued audio-grounded answer"]
    Direct --> Out

Key Designs

1. Branch Disagreement Gate: Preventing unwarranted interference on concordant distributions Under normal operating conditions, accompanying textual context is either harmoniously consistent with the audio or the model's joint branch is sufficiently robust to disregard minor textual noise. Indiscriminately mixing reference logits across all inputs risks destabilizing correct predictions, degrading conversational fluency and damaging base accuracy. To eliminate false triggers, GACL establishes a discrete sequence-level indicator \(N_{\text{out}}(x)\) acting as a gatekeeper: $\(N_{\text{out}}(x) = \mathbf{1}[\hat{y}_A \neq \emptyset] \cdot \mathbf{1}[\hat{y}_A \neq \hat{y}_J]\)$ where \(\hat{y}_J\) and \(\hat{y}_A\) represent the parsed top-1 candidate predictions from branches \(J\) and \(A\), respectively. The correction pipeline remains strictly dormant unless the same-audio reference yields a valid parse and explicitly contradicts the joint output. This mechanism ensures that 90.0% of free-form exact surface forms are pristine on faithful samples, bounding the risk of regression on non-conflict queries.

2. Reference Reliability Gate: Safeguarding against error amplification on noisy acoustic tasks In acoustically ambiguous or inherently difficult tasksโ€”such as subtle speech emotion recognition (SER)โ€”even the isolated audio reference branch can hallucinate or fail due to low signal-to-noise ratio. Blindly interpolating toward an erroneous reference would amplify noise rather than rescue truth. To resolve this, the method introduces a continuous reliability gate \(R_A(x)\) based on the reference branch's winning margin: $\(\Delta_A(x) = s_A(\hat{y}_A) - \max_{y \in \mathcal{H}(x), y \neq \hat{y}_A} s_A(y), \quad R_A(x) = \text{clip}\left(\frac{\Delta_A(x)}{\tau_A}, 0, 1\right)\)$ where \(s_A(y)\) denotes the candidate-normalized log-probability under the reference branch, \(\Delta_A(x)\) measures the margin separating the top candidate from the runner-up, and \(\tau_A\) is a temperature hyperparameter calibrating check strictness. When the reference branch exhibits low confidence, \(R_A(x)\) decays toward 0, suppressing correction; only when acoustic evidence yields a decisive margin does \(R_A(x)\) permit logit adjustment.

3. Bounded Convex Logit Interpolation: Constraining correction within valid solution manifolds Layer-wise activation patching demonstrates that the internal causal rescue vector in the late residual stream correlates almost perfectly with observable output score differences \(s_A - s_J\) (\(\rho = 0.93\)). However, this rank correlation certifies only direction, not unrestricted step magnitude. Allowing extrapolation (\(\alpha > 1\)) can propel generation outside the candidate manifold, precipitating unparsable tokens and syntax collapse. Therefore, GACL restricts logit updates to a bounded convex combination: $\(\alpha(x) = \text{clip}\left(\lambda \cdot R_A(x) \cdot N_{\text{out}}(x), 0, 1\right)\)$ $\(z_{k}^{\text{GACL}} = z_{J,k} + \alpha(x) \cdot (z_{A,k} - z_{J,k})\)$ where \(z_{J,k}\) and \(z_{A,k}\) represent next-token logits at decoding step \(k\) given the corrected prefix, and \(\lambda\) caps the maximum permissible shift. Restricting \(\alpha(x) \in [0, 1]\) guarantees that the adjusted representation stays along the line segment between the joint context and the isolated audio context, steering decision margins past zero without overshooting into out-of-distribution logit space.

A Worked Example

Consider an input containing a continuous 23-second siren sound \(x\), accompanied by the conflicting description "The sound is not constant." and the prompt "Is the sound constant? Please answer with yes or no.". Under the joint branch \(J\), the model is misled by the text, scoring \(s_J(\text{yes}) = -1.8\) and \(s_J(\text{no}) = -0.18\) over candidate set \(\mathcal{C}=\{\text{yes}, \text{no}\}\), yielding incorrect decision \(\hat{y}_J = \text{no}\) with margin \(M_J = -1.62\). Under the same-audio reference branch \(A\) where the text description is stripped, the acoustic evidence dominates, yielding \(s_A(\text{yes}) = -0.05\) and \(s_A(\text{no}) = -3.02\), with parsed prediction \(\hat{y}_A = \text{yes}\) and winning margin \(\Delta_A(x) = 2.97\). Because \(\hat{y}_A \neq \hat{y}_J\), the disagreement gate fires (\(N_{\text{out}}(x) = 1\)). Setting standard hyperparameters \(\tau_A = 0.5\) and \(\lambda = 0.5\), the reliability gate evaluates to \(R_A(x) = \text{clip}(2.97 / 0.5, 0, 1) = 1.0\). The bounded correction factor is computed as \(\alpha(x) = \text{clip}(0.5 \times 1.0 \times 1, 0, 1) = 0.5\). Applying the convex logit interpolation yields: $\(s^{\text{GACL}}(\text{yes}) = -1.8 + 0.5 \times (-0.05 - (-1.8)) = -0.925\)$ $\(s^{\text{GACL}}(\text{no}) = -0.18 + 0.5 \times (-3.02 - (-0.18)) = -1.600\)$ The corrected margin flips cleanly from \(-1.62\) to \(+0.675\), safely converting the top-1 prediction to the audio-supported truth "yes" without parameter retraining.

Key Experimental Results

Main Results

Evaluation is conducted along a rescueโ€“faithfulness frontier, measuring normalized area under the curve (nAUC) under strict (0โ€“5 percentage points faithful drop) and relaxed (0โ€“10 pp) budgets. Baselines include the uncorrected Joint branch, Audio-Aware Decoding (AAD, audio-removed reference), and Audio Contrastive Decoding (ACD, perturbed-audio reference).

Model Method AQA (0โ€“5 / 0โ€“10) VSC (0โ€“5 / 0โ€“10) SER (0โ€“5 / 0โ€“10) ALME (0โ€“5 / 0โ€“10) Average nAUC (0โ€“5 / 0โ€“10)
Qwen2-Audio-Inst. AAD 12.1 / 18.9 18.4 / 29.4 2.3 / 3.3 39.5 / 43.7 18.0 / 23.9
Qwen2-Audio-Inst. ACD 8.1 / 9.1 5.6 / 10.4 0.0 / 0.0 21.5 / 26.8 8.8 / 11.6
Qwen2-Audio-Inst. GACL (Ours) 51.0 / 60.7 39.1 / 51.2 2.9 / 6.1 60.4 / 63.4 38.3 / 45.4
Qwen2.5-Omni AAD 14.8 / 28.1 48.4 / 56.6 3.4 / 6.8 54.8 / 56.4 30.4 / 37.0
Qwen2.5-Omni ACD 0.5 / 0.5 29.4 / 37.0 3.9 / 5.7 32.8 / 41.2 16.7 / 21.1
Qwen2.5-Omni GACL (Ours) 31.6 / 45.7 58.0 / 71.6 7.6 / 11.4 66.0 / 66.0 40.8 / 48.7
Voxtral-Small AAD 0.0 / 2.8 7.1 / 8.5 2.8 / 4.4 17.9 / 29.4 7.0 / 11.3
Voxtral-Small ACD 1.7 / 3.1 0.5 / 0.5 2.3 / 2.9 2.1 / 7.0 1.6 / 3.4
Voxtral-Small GACL (Ours) 8.8 / 17.4 25.0 / 28.1 6.7 / 10.3 72.0 / 76.0 28.1 / 33.0
Qwen3-Omni AAD 11.8 / 20.1 7.0 / 14.8 0.0 / 0.0 44.7 / 45.0 15.9 / 20.0
Qwen3-Omni ACD 8.1 / 8.2 0.7 / 0.9 0.8 / 1.2 5.2 / 5.2 3.7 / 3.9
Qwen3-Omni GACL (Ours) 30.1 / 38.6 36.2 / 42.5 6.6 / 10.2 64.0 / 64.0 34.2 / 38.8
Kimi-Audio-Inst. AAD 10.5 / 17.2 12.3 / 24.8 0.5 / 0.7 2.0 / 4.2 6.3 / 11.8
Kimi-Audio-Inst. ACD 5.4 / 8.6 1.7 / 3.2 0.8 / 1.1 2.2 / 4.1 2.5 / 4.2
Kimi-Audio-Inst. GACL (Ours) 39.3 / 48.7 12.7 / 19.2 2.0 / 4.9 47.0 / 47.0 25.2 / 30.0

Applying GACL with fixed default parameters \((\lambda=0.5, \tau_A=0.5)\) directly to visionโ€“text arbitration on MC2 yields a +40.5 pp adversarial gain on Qwen3-VL-2B (from 43.0% to 83.5%) with 0.0 pp faithful drop, and +26.5 pp on Qwen3-VL-8B (from 55.5% to 82.0%) with only a 0.5 pp drop, highlighting robust cross-modal transferability.

Ablation Study

The ablation benchmarks assess the failure surfaces managed by each constituent component across the 5ร—4 model-task evaluation grid.

Variant MCQ@5pp Drop (โ†“) Low-RA Drop (โ†“) FF Surface Preservation (โ†‘) SER Parse Failure (โ†“) Avg. Conflict Gain (pp, โ†‘)
Joint (No correction) 0.0 0.0 100.0% 0.0% 0.0
Hard switch (Endpoint) 13.3 (Infeasible) โ€” โ€” โ€” 66.5
Naive interp. (Fixed \(\alpha=0.55\)) 4.6 4.7 70.0% 7.0% 38.4
w/o bound (\(\alpha \le \infty\)) 2.1 2.1 90.0% 27.0% 46.4
w/o \(N_{\text{out}}\) 2.1 2.1 22.0% 0.0% 46.1
w/o \(R_A\) 2.3 3.2 90.0% 0.0% 47.4
GACL (Full method) 2.1 2.1 90.0% 0.0% 45.8

Key Findings

  • Arbitration reversal localizes to the answer boundary: Negative controls injecting reference states into audio-span tokens or instruction-prompt tokens achieve 0.00 rescue across all models. Conversely, injecting hidden states exclusively at the answer boundary achieves an average rescue of 0.81 (reaching 0.98โ€“1.00 on AQA). Perceptual representations remain fully intact through intermediate layers; text dominance occurs during final decision consolidation.
  • Reference quality dictates the operational envelope: Decomposing rescue along the three sequential stages (opportunity \(\rightarrow\) gate activation \(\rightarrow\) top-1 conversion) reveals 66.3% / 99.3% / 90.9% rates across non-SER tasks. On SER, while opportunity (56.6%) and gate activation (94.2%) remain high, conversion drops to 48.6% because the isolated audio reference itself is accurate on only 35.3% of inputs. The reliability gate \(R_A\) properly dampens interventions when reference signals are weak.
  • Competitiveness with supervised fine-tuning: GACL captures 76% of the adversarial gain achieved by an in-domain LoRA baseline (retaining 93% on AQA and 85% on VSC) without parameter updates or risk of catastrophic forgetting, outperforming prompt engineering variants like audio-priority prompting by a 5ร— margin (+34.1 pp vs +6.7 pp at matched faithful drop).

Highlights & Insights

  • From evidence degradation to same-modality counterfactuals: Instead of asking how models react to degraded sensory inputs, the authors recognize that multimodal arbitration must be probed by asking which modality wins when sensory evidence is held constant and contextual noise is removed.
  • Bridging internal mechanistic insight to output decoding: By rigorously establishing a 0.93 Spearman correlation between internal residual state updates and observable final score differences, the paper bypasses the need for internal tensor manipulation or runtime backward passes.
  • Zero-parameter engineering efficiency: Leveraging audio encoder prefix caching allows dual-branch evaluation with only a 1.27ร— inference latency overhead and negligible GPU memory increment.

Limitations & Future Work

  • Ceiling capped by reference decodability: GACL rescues arbitration failures, not fundamental perceptual blind spots. If the raw audio is excessively distorted or unintelligible, the reference branch cannot provide a valid donor signal.
  • White-box model dependency: Computing token-level logit interpolation requires access to raw probability distributions, precluding deployment behind text-only black-box commercial APIs.
  • Scaling to complex multimodal contexts: Benchmark conflicts examined here are structured sentences paired with short audio clips; adapting the framework to long-form retrieval-augmented documents or multi-speaker overlapping audio remains open for future research.
  • vs AAD / ACD (Audio Contrastive Decoding): AAD and ACD perturb or drop audio to identify multimodal hallucinations; GACL holds audio constant and drops textual interference, targeting evidence arbitration rather than modality bias.
  • vs VCD / OPERA / ICD (Vision Contrastive Decoding): Standard vision-language contrastive decoding tackles object hallucinations rather than explicit contradictory assertions; GACL shows that counterfactual logit correction seamlessly generalizes across modalities to visual-text conflicts without parameter tuning.
  • vs Supervised Fine-Tuning (SFT / LoRA): SFT requires labor-intensive conflict curation and risks eroding general instruction-following; GACL delivers competitive conflict resilience training-free at inference time.

Rating

  • Novelty: โญโญโญโญโญ [Pioneering identification of repairable arbitration reversals and formulation of same-audio counterfactual decoding]
  • Experimental Thoroughness: โญโญโญโญโญ [Exhaustive evaluation across 5 open ALMs, 4 conflict tasks, layer-wise activation patching, and cross-modal transfer]
  • Writing Quality: โญโญโญโญโญ [Extremely rigorous narrative moving cohesively from behavioral diagnostics to mechanistic proof and practical decoding rules]
  • Value: โญโญโญโญโญ [Provides a practical, plug-and-play solution for trustworthy multimodal decision-making under evidence conflict]