Skip to content

Modality-Specific Gating for Large Audio-Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Large Audio-Language Models, Modality-Specific Gating, Modality Bias, Cross-Modal Conflict, Robustness

TL;DR

To alleviate the persistent problem where Large Audio-Language Models (LALMs) over-rely on textual prompts and ignore acoustic evidence during autoregressive decoding, this work introduces Modality-Specific Gating (MSG), a parameter-efficient mechanism that injects query-conditioned dynamic biases into audio attention logits before softmax to substantially improve acoustic grounding under adversarial text prompts.

Background & Motivation

Large audio-language models (LALMs) have transformed auditory intelligence by projecting acoustic features extracted from pretrained audio encoders into the shared token space of large language models (LLMs). This end-to-end integration empowers models with remarkable general audio understanding, spoken dialogue capabilities, and zero-shot instruction following. However, because current LALMs rely fundamentally on text-conditioned autoregressive generation architectures, the native compatibility of textual representations with the foundational language model backbone inherently induces an asymmetric competitive advantage for text tokens over projected acoustic tokens within the causal self-attention pool.

When user prompts contain misleading cues, hallucinations, or uninformative semantic noise that conflicts with the actual acoustic audio signal, this architectural imbalance manifests as a severe language-dominant modality bias. By quantitatively probing Modality-Specific Attention (MSA) distributions across decoding layers and generation steps in models like Qwen2-Audio, empirical evidence reveals a chronic audio-text imbalance: text-related tokens systematically consume the vast majority of the attention budget across both shallow and deep layers. Furthermore, as autoregressive generation progresses and the generated textual history expands, the attention share allocated to acoustic keys is severely compressed. Once early generation steps drift toward textual priors, the subsequent hidden states become increasingly language-centric, establishing a self-amplifying feedback loop that permanently suppresses acoustic grounding.

While modality bias mitigation has received substantial attention in vision-language tasks through causal debiasing or multi-stage pipelines, existing attempts in audio-language models predominantly rely on static, training-free inference-time attention reweighting heuristics that fail to adjust across dynamic query contexts. This paper attacks the issue by transforming the unconstrained token competition inside self-attention into a query-aware regulated gating process. Core idea: propose Modality-Specific Gating (MSG), an acoustic calibration mechanism that computes a dynamic query-conditioned gating scalar from layer hidden states and injects an additive bias specifically onto the pre-softmax attention logits of audio keys, efficiently restoring acoustic evidence visibility and severing text-driven feedback loops with minimal parameter overhead.

Method

Overall Architecture

The MSG framework is engineered to restore audio-text attention balance during autoregressive generation without altering the underlying Transformer backbone architecture or disturbing standard dot-product attention mechanics. Given an audio recording and a textual instruction, acoustic features are projected and concatenated with text token embeddings before being processed by the causal Transformer decoder. At each decoding step within every decoder layer, MSG computes a dynamic query-conditioned gating scalar from the current layer hidden state, scales it by a global learnable scalar, and applies an additive bias exclusively to the pre-softmax attention logits corresponding to audio key positions before executing standard normalized attention.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Audio & Text Input<br/>Acoustic projection & text concatenation"] --> B["Modality Attention Probing & Group Identification<br/>Identify acoustic key positions maudio"]
    B --> C["Query Hidden State Projection & Dynamic Gating<br/>Compute step-wise gate rt = σ(W ht)"]
    C --> D["Global Scaling & Adaptive Bias Calibration<br/>Formulate additive bias Bt = β · rt · maudio"]
    D --> E["Logit-Level Attention Modulation & Generation<br/>Softmax(QK^T / √dk + Bt) V"]

Key Designs

1. Modality Attention Probing & Group Identification: Quantifying audio attention dilution and indexing acoustic key masks Under autoregressive decoding with unified multimodal prefixes, audio tokens and prompt tokens directly compete under a shared softmax denominator, causing acoustic signals to be marginalized by extensive textual prompts. To implement targeted calibration without invasive modifications, the system partitions source sequence keys into modality groups. Utilizing the unique audio boundary tokens (such as <|AUDIO|>) present in the input stream, the model identifies key indices belonging to acoustic representations and constructs a time-invariant binary indicator vector \(m_{\text{audio}} \in \{0, 1\}^K\). This provides an exact, computation-free positional mask isolating acoustic keys from instructions, template tokens, and generated histories.

2. Query Hidden State Projection & Dynamic Gating: Adapting acoustic attention boost to step-specific semantic needs Different generation steps exhibit vastly disparate dependencies on acoustic evidence: predicting emotional states or sound categories relies entirely on acoustic signals, whereas outputting syntax connectors or template formatting words requires adherence to linguistic priors. Applying an invariant boost would corrupt textual fluency. To resolve this, MSG introduces a lightweight linear projection vector \(W \in \mathbb{R}^{1 \times d_k}\) shared across all decoder layers, deriving a step-wise bounded gating score from the current query hidden state \(h_t \in \mathbb{R}^{d_k}\): $\(r_t = \sigma(W h_t) \in (0, 1)\)$ This query-conditioned signal dynamically assesses whether the immediate generation step necessitates acoustic verification, adaptively opening the acoustic attention pathway only when critical decisions are formed.

3. Global Scaling & Adaptive Bias Calibration: Decoupling global modulation magnitude from step dynamics Relying solely on an unconstrained gating activation risks unstable logit scaling across diverse evaluation samples, whereas altering normalized weights directly compromises pretrained representation spaces. MSG introduces a single positive learnable scalar \(\beta\) to decouple overall magnitude calibration from token-wise dynamic gating. For decoding step \(t\), the additive bias row is parameterized as: $\(B_{t, :} = \beta \cdot r_t \cdot m_{\text{audio}}\)$ This vector is directly incorporated into the pre-softmax attention calculation: $\(\text{Attention}_t(Q, K, V) = \text{softmax}\left(\frac{q_t K^\top}{\sqrt{d_k}} + \beta \cdot r_t \cdot m_{\text{audio}}\right) V\)$ By applying the calibration strictly at the pre-softmax logit level, MSG fully preserves standard self-attention normalization and gradient properties while preventing early text dominance from recursively suppressing subsequent acoustic attention.

Loss & Training

MSG is optimized using parameter-efficient fine-tuning (PEFT) while keeping the base audio encoder and LLM backbone completely frozen. Low-Rank Adaptation (LoRA) is inserted exclusively into the text self-attention projections (rank \(r=8\), \(\alpha=16\), dropout \(0.05\)). For Qwen2-Audio-7B-Instruct, the total trainable footprint is only 10.35M parameters (~0.12% of the 8.41B base model).

To prevent the acoustic gate from collapsing to zero during early training stages, a hinge-style gate regularization objective is introduced: $\(\mathcal{L}_{\text{gate}} = \max(0, \tau - \bar{r})\)$ where \(\bar{r}\) represents the mean acoustic gate activation across answer tokens and \(\tau\) denotes an automatically calibrated gate activation threshold (empirically set around \(0.65\)). The regularizer acts as a one-sided lower bound: once gate activations satisfy the threshold, the penalty vanishes, allowing task cross-entropy \(\mathcal{L}_{\text{task}}\) to guide unhindered convergence. Optimization uses AdamW with a learning rate of \(5\times 10^{-5}\), effective batch size 16 (gradient accumulation 8), and mixed-precision on an NVIDIA RTX 5090 GPU.

Key Experimental Results

Main Results

Evaluation was conducted on Vocal Sound Classification (VSC on VocalSound with 3,594 test audio clips) and Speech Emotion Recognition (SER on MELD with 2,610 test utterances), benchmarking against specialized audio baselines, proprietary foundation models (GPT-4o-Audio, Gemini-3-Flash), and open-source models.

Dataset / Task Metric Qwen2-Audio + MSG (Ours) Qwen2-Audio (Base) Gain Benchmark References
VocalSound (VSC) ACC (%) ↑ 94.7 94.0 +0.7 Gemini-3-Flash: 88.2 / GPT-4o-Audio: 82.5
VocalSound (VSC) ACC (%) ↑ (Qwen2.5-Omni) 94.2 93.9 +0.3 CLAP: 49.4 / Pengi: 60.4
MELD (SER) WA (%) ↑ 50.4 37.6 +12.8 WavLM Large: 54.2 / Vesper-12: 52.5
MELD (SER) UA (%) ↑ 39.0 36.2 +2.8 GPT-4o-Audio: 42.9 / emotion2vec: 28.0
MELD (SER) WF1 (%) ↑ 50.9 39.3 +11.6 Qwen2.5-Omni: 48.9 / Gemini-3-Flash: 31.3
MELD (SER) (Omni) WA / UA / WF1 (%) ↑ 57.4 / 39.1 / 55.8 50.4 / 32.8 / 48.9 +7.0 / +6.3 / +6.9 MATA (Omni): 50.4 / 30.9 / 47.7

Robustness stress-testing across prompt variations on MCR-Bench demonstrates remarkable resistance against misleading text prompts:

Model Backbone VSC Neutral (Neu) VSC Faithful (Fai) VSC Adversarial (Adv) VSC Irrelevant (Irr) SER Neutral (Neu) SER Adversarial (Adv) SER Irrelevant (Irr)
Qwen2-Audio-7B-Instruct 0.854 0.998 0.118 0.857 0.418 0.000 0.278
Qwen2-Audio + MATA 0.853 0.998 0.122 0.852 0.426 0.000 0.277
Qwen2-Audio + MSG (Ours) 0.947 0.967 0.929 0.948 0.466 0.212 0.473
Qwen2.5-Omni-7B 0.903 0.991 0.058 0.827 0.480 0.000 0.405
Qwen2.5-Omni + MSG (Ours) 0.940 0.963 0.917 0.932 0.496 0.227 0.472

Ablation Study

Detailed component ablations on MCR-Bench-VSC using Qwen2-Audio isolate the individual effects of supervised fine-tuning (SFT), the global scaling scalar \(\beta\), and the dynamic token-wise gate \(r\):

Model Configuration Neutral ACC↑ Faithful ACC↑ / TIR↑ Adversarial ACC↑ / TIR↓ Irrelevant ACC↑ / TIR↓ Macro ACC↑ Modal Robustness Score (MRS)↑
Base (w/o SFT, w/o MSG) 0.854 0.998 / 0.144 0.118 / 0.736 0.857 / 0.099 0.658 0.311
SFT-only (w/o MSG) 0.929 0.983 / 0.054 0.610 / 0.319 0.911 / 0.030 0.835 0.721
w/o \(r\) (global-\(\beta\) only) 0.944 0.990 / 0.046 0.538 / 0.407 0.921 / 0.029 0.816 0.651
w/o \(\beta\) (token-wise \(r\) only) 0.935 0.971 / 0.036 0.905 / 0.040 0.934 / 0.021 0.937 0.974
Full Model (MSG w/ \(\beta + r\)) 0.947 0.967 / 0.020 0.929 / 0.021 0.948 / 0.013 0.948 0.985

Key Findings

  • Dynamic gating \(r\) is the cornerstone of adversarial robustness: Ablating \(r\) causes adversarial accuracy to collapse from 92.9% to 53.8%, while the Text Influence Rate (TIR) surges to 0.407. In contrast, incorporating \(r\) alone achieves 90.5% adversarial accuracy, confirming that query-dependent gating suppresses spurious textual cues.
  • Global scalar \(\beta\) ensures stable probability calibration: Coupling \(\beta\) with \(r\) delivers the optimal trade-off, attaining peak Macro accuracy (0.948) and the highest Modal Robustness Score (0.985) while keeping adversarial prediction flipping minimal (TIR = 0.021).
  • A measured robustness-faithfulness trade-off: Under faithful prompts that align with acoustic evidence, suppressing reliance on linguistic shortcuts causes a mild accuracy retreat (from 0.998 to 0.967 on VSC, and from 1.000 to 0.847 on SER). Nonetheless, faithful accuracy remains substantially higher than neutral accuracy, proving that beneficial textual guidance remains actively leveraged.

Highlights & Insights

  • Parameter-efficient logit-level intervention: MSG requires zero structural modifications to Transformer attention blocks, introducing only a single projection vector and a global scalar (~0.12% parameters with LoRA) to seamlessly resolve cross-modal attention starvation.
  • Breaking the autoregressive feedback loop: The analysis uncovers that modality bias in LALMs is an compounding failure mode where early text-biased generations progressively starve acoustic keys of attention in later steps. MSG effectively breaks this self-reinforcing dynamic at its root.
  • Seamless generalization to omni-modal architectures: Beyond speech-specialized models, MSG transfers seamlessly to unified omni-modal foundations like Qwen2.5-Omni, demonstrating broad applicability across multimodal generative architectures.

Limitations & Future Work

  • Evaluation restricted to short-span classification tasks: Experiments center on vocal sound classification and speech emotion recognition; validation on long-form audio reasoning, multi-speaker conversational reasoning, and open-ended audio QA remains open for future exploration.
  • Natural skew in affective speech distributions: Benchmarks like MELD exhibit heavy class imbalance, leaving minority emotion categories susceptible to dataset-level sparsity.
  • Layer-shared gating constraints: The default implementation shares projection weights \(W\) across all decoder layers. While parameter-efficient, appendix results show that layer-specific projections achieve further accuracy improvements (e.g., reaching 60.2% WA on MELD), suggesting promising directions for multi-scale hierarchical gating.
  • vs MATA (Wang et al., 2025b): While MATA performs training-free static attention reweighting during inference, it cannot adapt to dynamic query contexts across generation steps; MSG trains adaptive query gating, outperforming MATA by over 80 percentage points under adversarial MCR-Bench settings (92.9% vs 12.2%).
  • vs Visual Debiasing (RUBi / CF-VQA): Vision-language approaches typically construct complex counterfactual causal branches or multi-stage unimodal teacher networks; MSG operates directly on the core autoregressive attention mechanism, offering a vastly cleaner and more elegant integration.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Elegantly formulates a query-conditioned logit bias that directly addresses the step-wise attention starvation of acoustic tokens in LALMs.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation covering standard benchmarks, four-condition MCR-Bench stress tests, component ablations, attention visualizations, and multi-seed validation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative progression connecting empirical attention probing to architectural gating formulations.
  • Value: ⭐⭐⭐⭐⭐ Provides an effective, lightweight, and deployment-friendly solution for resolving prompt-induced hallucinations in multimodal audio systems.