DiffA: Large Language Diffusion Models Can Listen and Understand

Conference: AAAI 2026
arXiv: 2507.18452
Code: GitHub
Area: Image Generation
Keywords: Diffusion Language Models, Audio Understanding, Large Audio-Language Models, LLaDA, Parameter-Efficient Adaptation

TL;DR

Proposed DIFFA, the first Large Audio-Language Model based on a diffusion language model. By using a frozen LLaDA-8B backbone, a lightweight dual-adapter architecture, and a two-stage training pipeline, it achieves performance competitive with autoregressive baselines on MMSU, MMAU, and VoiceBench using only 960 hours of ASR data and 127 hours of synthetic instruction data.

Background & Motivation

Large Audio-Language Models (LALMs) have developed rapidly in recent years, but existing methods are almost entirely based on the Autoregressive (AR) decoding paradigm:

Encoder + LLM Paradigm (e.g., Qwen2-Audio, SALMONN): Projects the speech encoder output into the LLM input space via adapters.

Speech Tokenization Paradigm (e.g., SpeechGPT, Moshi): Discretizes audio into tokens and directly trains them using an LLM.

Both paradigms rely on AR decoding, which exhibits inherent drawbacks: exposure bias, slow generation speed, lack of bidirectional contextual modeling, and limited flexibility for partial conditional inference.

Diffusion language models (such as LLaDA) have demonstrated capabilities comparable to AR models in the text domain and have been extended to vision-language tasks (LLaDA-V). However, the audio modality remains entirely unexplored within diffusion language models. Given the unique acoustic variability, complex temporal structure, and rich paralinguistic information of audio, whether it is suitable for diffusion modeling remains an open question.

Method

Overall Architecture

DIFFA adopts a modular and efficient design:

• Speech Encoder: Frozen Whisper-Small (88.2M parameters)
• Dual Adapters: Semantic Adapter (14.4M) + Acoustic Adapter (22.3M)
• Language Backbone: Frozen LLaDA-8B-Instruct (8.1B parameters)

The encoder and language model remain frozen throughout training. Only the two lightweight adapters are trained, with total trained parameters of approximately 36.7M (< 0.5%).

Key Designs

1. Dual-Adapter Architecture

Semantic Adapter: - 2-layer convolutional network (4× downsampling) + 2-layer linear projection - Compresses Whisper's 50 Hz output to 12.5 Hz - Extracts high-level semantic features from the final output of the encoder

Acoustic Adapter: - 2-layer Q-Former structure with 64 trainable query vectors - Extracts low-level acoustic features (pitch, emotion, speech rate, and other paralinguistic information) from the intermediate layer states of the encoder

The final audio representation is a concatenation of the outputs from both adapters, which is appended to the LLM input as prefix tokens.

2. LLaDA-Based Diffusion Decoding

LLaDA defines a forward masking process: each token is replaced with a special mask token \(\text{M}\) with probability \(t\). The mask predictor \(p_\theta(x_0|x_t)\) is trained to recover the masked tokens:

\[\mathcal{L}(\theta) = -\mathbb{E}_{t,x_0,x_t}\left[\frac{1}{t}\sum_{i=1}^{L} \mathbf{1}[x_t^i = \text{M}] \log p_\theta(x_0^i|x_t)\right]\]

After incorporating audio conditioning, the objective function becomes:

\[L_a = -\mathbb{E}_{t,a_0,p_0,r_0,r_t}\left[\frac{1}{t}\sum_{i=1}^{L'} \mathbf{1}[r_t^i = \text{M}] \log p_\theta(r_0^i \mid a_0, p_0, r_t)\right]\]

During training, the audio and prompt tokens remain unmasked, and the mask-and-predict operation is performed only on the response tokens.

3. Data Construction

Inspired by the DESTA series, synthetic instruction data is generated by prompting LLMs: - Inputs: Audio transcripts + acoustic attributes (10 annotated attributes including gender, accent, emotion, duration, etc.) - Prompt: "What can you hear from the audio?" - The response generated by the LLM serves as the supervision signal.

In addition, self-distillation rewriting is introduced: initial descriptions are first generated using Qwen3-8B and then rewritten by LLaDA to align with its internal data distribution.

Loss & Training

Two-Stage Training:

Stage 1—Semantic Alignment: - Data: LibriSpeech 960 hours - Objective: ASR task, training the semantic adapter to align the speech encoder with the language model - Only the semantic adapter is trained

Stage 2—Modality Alignment: - Data: 127 hours of synthetic instruction data (VCTK, Accentdb, IEMOCAP, dailytalk, VoxCeleb1) - Objective: Audio captioning task, training the dual adapters - Both the semantic adapter and acoustic adapter are trained simultaneously

Inference adopts a semi-autoregressive strategy: generation is performed block-by-block from left to right, predicting tokens within blocks in parallel and re-masking low-confidence positions.

Key Experimental Results

Main Results

Table 2: MMSU Benchmark (Speech Language Understanding, 5000 samples)

Model	Perception	Reasoning	Overall
Gemini-1.5-Pro	46.10	76.16	60.68
Qwen2.5-Omni	42.50	79.83	60.57
Qwen2-Audio-Instruct	39.02	68.90	53.27
DIFFA	40.28	72.92	56.04

Table 3: MMAU Benchmark (Audio Reasoning, 3 Domains, 27 Skills)

Model	Sound	Music	Speech	Avg
GPT-4o mini Audio	50.75	39.22	69.07	53.01
Qwen2-Audio-Instruct	67.27	56.29	55.26	59.61
DIFFA	46.25	43.41	59.46	49.71

Table 4: VoiceBench (Speech QA)

Model	SD-QA	OBQA	IFEval	AdvBench	Overall
Qwen2-Audio	35.72	49.45	26.33	96.73	55.34
DiVA	57.06	25.49	39.16	98.27	55.70
DIFFA	34.45	35.60	26.56	76.54	48.22

DIFFA outperforms Qwen2-Audio-Instruct by nearly 3 points on MMSU, with semantic reasoning being particularly outstanding (81.53%).

Ablation Study

Table 5: Ablation on Architecture and Adapter Design

LLM Backbone	Adapter	MMAU	MMSU
LLaMA 3.1 (AR)	Dual	28.40	38.40
LLaDA (Diffusion)	Single	47.70	52.88
LLaDA (Diffusion)	Dual	49.71	56.04

Table 6: Ablation on Instruction Data Sources

Data Source	MMAU	MMSU	VoiceBench	Avg
LLaMA 3	51.71	54.72	37.17	47.86
Qwen3	49.71	56.04	48.22	51.32
rewrite-Qwen3	50.41	56.43	46.60	51.15

Key Findings

1. Diffusion vs. Autoregression: Replacing the AR backbone (LLaMA 3.1) with the diffusion backbone (LLaDA) yields a 23.3-point gain on MMAU (28.4 \(\rightarrow\) 51.7) and a 16.3-point gain on MMSU, demonstrating the clear superiority of diffusion language models for audio understanding.
2. Dual-Adapter > Single-Adapter: The acoustic adapter brings gains of +2.01 (MMAU) and +3.16 (MMSU), proving the importance of low-level acoustic features.
3. High Data Efficiency: High competitive performance is achieved with only 127 hours of synthetic data + 960 hours of ASR data, compared to the 510,000 hours of training data used by Qwen2-Audio.
4. DIFFA performs best on semantic reasoning tasks (81.53%), while its phonological and paralinguistic perception is relatively weaker.
5. Instruction data generated with LLaDA outperforms that generated with LLaMA-3, suggesting that the inductive biases of diffusion models might produce more aligned supervision signals.

Highlights & Insights

1. First Diffusion-Based LALM: Opens up a new direction in audio understanding, proving that diffusion language models are highly viable not only for text and vision but also for audio.
2. Extreme Data Efficiency: Competes effectively while using less than 1/400 of the training data compared to Qwen2-Audio, requiring only 72 A800 GPU hours.
3. Elegant Freezing Strategy: Freezing both the encoder and the language backbone while training only 36.7M adapter parameters (< 0.5%) effectively avoids catastrophic forgetting.
4. Dual-Adapter Design: The separate semantic and acoustic pathways complement each other, capturing high-level semantics and low-level paralinguistic information respectively.

Limitations & Future Work

1. Limited Training Data: The 960 + 127 hours of data may restrict generalization performance in complex scenarios such as low-resource accents and noisy environments.
2. Insufficient Paralinguistic Perception: Performance on phonological and paralinguistic tasks is significantly lower than on semantic reasoning, indicating a need for more acoustic training data.
3. Inference Latency: Diffusion decoding requires multi-step iterations, meaning the actual inference latency could be higher than that of AR models.
4. Lack of Speech Generation Capability: Currently designed solely for understanding rather than generation, which limits its functionality compared to full-duplex conversational systems (e.g., Moshi).
5. Potential for extending training data to cover a wider range of audio types (e.g., music and environmental sounds).

Related Work & Insights

• LLaDA: The foundation for diffusion language models, proving that a masked diffusion process can replace autoregression.
• LLaDA-V: Extends diffusion LLMs to vision-language tasks, which this work further extends into the audio domain.
• DESTA-2: A construction paradigm for synthetic instruction data, but utilizing a cascaded design (Whisper transcription followed by LLM generation). This work implements an end-to-end paradigm.
• Qwen2-Audio: A primary competitor with robust performance, though it demands immense volumes of training data (510,000 hours).
• Inspiration: The modality expansion paradigm of diffusion LLMs—employing a frozen backbone and lightweight adapters—could emerge as a general solution for multimodal scaling.

Rating

• Novelty: ⭐⭐⭐⭐⭐ (First diffusion-based audio-language model)
• Technical Depth: ⭐⭐⭐⭐ (Well-designed dual-adapter + two-stage training)
• Experimental Thoroughness: ⭐⭐⭐⭐ (Evaluated on 3 benchmarks + comprehensive ablation)
• Value: ⭐⭐⭐⭐ (Extremely high data efficiency, beneficial for low-resource scenarios)
• Overall Score: 8.5/10

Related Papers

• [ACL 2026] SpeakerSleuth: Can Large Audio-Language Models Judge Speaker Consistency across Multi-turn Dialogues?
• [ACL 2025] Who Can Withstand Chat-Audio Attacks? An Evaluation Benchmark for Large Audio-Language Models
• [AAAI 2026] AHAMask: Reliable Task Specification for Large Audio Language Models without Instructions
• [AAAI 2026] Listening Between the Frames: Bridging Temporal Gaps in Large Audio-Language Models
• [ICML 2026] Focus Then Listen: An Empirical Study of Plug-and-Play Audio Enhancer for Noise-Robust Large Audio Language Models