Skip to content

AHAMask: Reliable Task Specification for Large Audio Language Models without Instructions

Conference: AAAI 2026 arXiv: 2509.01787v3 Code: https://github.com/X-LANCE/SALMONN-AHAMask Area: Speech & Audio Processing / Large Language Model Interpretability Keywords: Large Audio Language Models, Attention Head Masking, Prompt Sensitivity, Task Specification, Functional Pathways

TL;DR

By applying binary masks (AHAMask) over attention heads in the Transformer backbone of Large Audio Language Models (LALMs), specific acoustic task functionalities can be reliably triggered without any textual instructions, while revealing the existence of "acoustic functional pathways" within LALMs.

Background & Motivation

Current LALMs (e.g., SALMONN, Qwen2Audio) are capable of handling diverse audio tasks (ASR, emotion recognition, speaker verification, etc.) in a unified framework, but rely heavily on natural language instructions for task specification. The key issue is that semantically equivalent instructions—differing only in wording, punctuation, or capitalization—can cause severe performance fluctuations (e.g., WER on SALMONN's ASR task jumping from 2% to 12%). This prompt sensitivity renders LALMs unreliable in real-world deployment. Meanwhile, prior work in the text LLM domain (han2025heads) has demonstrated that attention head masking can trigger specific text tasks without instructions, yet this property remains unexplored in multimodal audio models.

Core Problem

How can the dependence and sensitivity of LALMs on natural language instructions be eliminated, enabling reliable acoustic task specification without any instructions? More fundamentally, do the Transformer attention heads in LALMs harbor "functional pathways" analogous to those found in text LLMs?

Method

Overall Architecture

AHAMask introduces a binary mask \(m_{i,j} \in \{0,1\}\) for each attention head in the decoder-only LLM backbone of a LALM. During inference, only the selected subset of attention heads is activated, modifying MHA as \(\text{MHA}_i(\mathbf{X}, \mathcal{M}) = \sum_{j=1}^{h} m_{i,j} \mathbf{Y}^{(i,j)} \mathbf{W}_O^{(i,j)}\). Due to skip connections, masking all heads in a given layer does not break the computation graph.

Key Designs

  1. Gumbel-Sigmoid Training: Since the mask \(\mathcal{M}\) is discrete, Gumbel-Sigmoid is used for gradient estimation. During training, a soft mask is computed as \(\mathbf{S} = \sigma((\mathbf{M} + \mathbf{G})/\tau)\), binarized via \(\mathcal{M} = \mathbb{I}(\mathbf{S} \geq 0.5)\), with the straight-through estimator (STE) used for backpropagation. The temperature \(\tau\) is linearly annealed from 4.0 to 0.5. The only trainable parameters are the mask logits \(\mathbf{M} \in \mathbb{R}^{n \times h}\), with a count equal to the total number of attention heads (e.g., only 1,600 parameters for SALMONN).
  2. Instruction-Free Training Paradigm: Training is conducted on specific downstream tasks using only audio-text pairs \((Audio_k, Text_k)\) without any instructions. Standard cross-entropy loss is applied for next-token prediction, with all original LALM parameters frozen.
  3. Sparsity Penalty: An optional regularization term \(\mathcal{L} = \mathcal{L}_{CE} + \lambda \sum_{i,j} m_{i,j}\) further reduces the number of active heads. Experiments show that with \(\lambda = 10^{-4}\), only 299 out of 1,600 heads are required to achieve 98.02% accuracy on the GR task.

Loss & Training

  • Loss: Standard cross-entropy \(\mathcal{L}_{CE}\) (next-token prediction), with optional L1 sparsity penalty
  • All head logits initialized from \(\mathcal{N}(4, 0.02)\) (all heads active at initialization)
  • Learning rate warmed up to 1e-2, then cosine decayed to 1e-4
  • Single-GPU training (65G Ascend 910B NPU)

Key Experimental Results

Dataset/Task Metric AHAMask (no instruction) With Instruction Notes
LibriSpeech ASR (SALMONN) WER 2.10/5.08 2.10/4.95 Nearly on par
GR (SALMONN) ACC 98.05% 96.79% AHAMask superior
SER (SALMONN) ACC 70.02% 69.70% AHAMask slightly better
ASV (SALMONN) ACC 93.24% 93.49% Nearly on par
AAC (SALMONN) METEOR/ROUGE-L 24.15/48.71 20.60/40.42 AHAMask significantly better
GR (Qwen2Audio-Instruct) ACC 94.43% 91.03% AHAMask +3.4%
ASV (Qwen2Audio-Base) ACC 85.75% 49.24% AHAMask +36.5%!
Composite GR|ASR (SALMONN) IFR / ACC / WER 99.12/97.77/2.21 98.59/68.02/3.52 AHAMask dominant across all metrics
Composite JSON format (SALMONN) IFR / WER / ACC 98.89/2.40/97.30 69.16/6.17/51.05 Instruction-based method completely fails

Ablation Study

  • Random masks are ineffective: Random masks with the same head count completely fail, confirming that specific head locations are critical.
  • Non-transferability across models: The AHAMask trained on Qwen2Audio-Instruct is entirely ineffective on the Base model, and vice versa.
  • Head count correlates with task complexity: Classification tasks (GR/SER/ASV) require fewer heads; sequence generation tasks require more.
  • Mask similarity reflects task relationships: OSR and ASR exhibit the highest Jaccard similarity, validating linguistic intuition.
  • "Many roads lead to Rome" effect: Masks trained with different random seeds differ by more than 30% yet achieve nearly identical performance; taking their intersection yields even fewer heads without performance degradation.
  • Functional pathways emerge progressively: Incrementally activating heads by importance weight yields smooth performance improvement rather than abrupt jumps.
  • Out-of-domain generalization: The GR task generalizes well across TEDLIUM, CommonVoice, and VoxCeleb1 (ACC 89–98%); ASR generalization requires more diverse training data.

Highlights & Insights

  • Extremely parameter-efficient: trainable parameters equal only the number of attention heads (1,600 for SALMONN, roughly 200 bytes of storage), orders of magnitude fewer than PEFT methods such as LoRA.
  • Inference is low-cost—binary masking actually reduces computation by deactivating a subset of heads.
  • Overwhelming advantage over instruction-based methods on composite tasks, particularly in instruction-following rate (IFR).
  • Reveals the existence of "acoustic functional pathways" in LALMs, an interesting interpretability finding.
  • Demonstrates that even base models (without instruction fine-tuning) can, via AHAMask, match or surpass instruct models.

Limitations & Future Work

  • Out-of-domain generalization for ASR remains limited; masks trained on a single domain may capture overly fine-grained features.
  • Composability is only validated for ASR+GR; combinations of three or more tasks remain unexplored.
  • Mask composability via Boolean operations across tasks has only been preliminarily explored.
  • A text-to-mask converter—automatically mapping natural language instructions to attention head masks—has not been investigated.
  • Validation is limited to three LALMs; larger-scale models (e.g., LLaMA-70B-scale audio models) are not covered.
  • Generative audio tasks (TTS, audio generation) are not explored.
  • han2025heads (Heads Are All You Need): This work directly extends the text LLM findings to the multimodal audio domain, confirming the existence of acoustic functional pathways. The core distinction lies in generalizing from pure text to audio-text multimodal alignment scenarios.
  • LoRA and other PEFT methods: LoRA requires millions of trainable parameters and maintains or increases parameter count at inference; AHAMask requires only thousands of parameters and actually reduces computation at inference.
  • Steering Vectors / Representation Engineering: These approaches control model behavior by adding directional vectors in the activation space, but still rely on instructions or incur additional inference overhead; AHAMask directly selects functional sub-networks at the structural level.
  • Multimodal functional pathway analysis: This finding could be extended to vision-language models (VLMs)—do attention heads in VLMs harbor analogous "visual functional pathways"? If so, could head masking enable instruction-free visual task specification?
  • Cross-task mask composition: Intersection and union operations over task-specific masks suggest a new paradigm for model editing and functional composition—combining capabilities purely through sub-network selection without modifying parameters.
  • Implications for model compression: AHAMask reveals that a large fraction of attention heads are redundant for specific tasks (e.g., SALMONN requires only 1/5 of heads for GR), providing task-aware guidance signals for structured pruning.
  • Text-to-mask systems: A promising future direction is training a lightweight network to map natural language instructions to attention head masks, combining the flexibility of instruction-based approaches with the reliability of masking.

Rating

  • Novelty: ⭐⭐⭐⭐ (Core idea extends han2025heads; innovation lies in multimodal generalization and in-depth analysis)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (3 models, 7+ tasks, composite tasks, ablations, generalization, visualization—highly comprehensive)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Clear structure, progressively organized experiments, deep analysis)
  • Value: ⭐⭐⭐⭐ (Reveals important interpretability findings, though practical applicability remains constrained by the need for task-specific training)