AAAI2026 Audio & Speech AI paper notes paper summaries Speech & Audio Sentiment Analysis Multimodal/VLM Dialogue Diffusion Models LLM

🎵 Audio & Speech¶

🤖 AAAI2026 · 30 paper notes

📌 Same area in other venues: 📷 CVPR2026 (22) · 🔬 ICLR2026 (80) · 💬 ACL2026 (72) · 🧪 ICML2026 (36) · 🧠 NeurIPS2025 (47) · 📹 ICCV2025 (11)

🔥 Top topics: Speech & Audio ×13 · Sentiment Analysis ×7 · Multimodal/VLM ×4 · Dialogue ×2 · Diffusion Models ×2

A Mind Cannot Be Smeared Across Time: This paper formally proves that whether a machine possesses consciousness depends not only on what it computes, but also when it computes. Strictly sequential execution systems do not satisfy the temporal co-instantiation condition required for the unity of consciousness, rendering pure software consciousness impossible on strictly sequential hardware.
DeepDebater: A Superpersuasive Autonomous Policy Debating System: This paper proposes DeepDebater, the first autonomous multi-agent system capable of participating in and winning a full policy debate tournament (eight speech rounds + cross-examinations). It utilizes a hierarchical agent workflow division of labor to construct Affirmative (Advantage) and Negative (DA + CP + Kritik) arguments. Supported by retrieval augmentation on over 3 million evidence cards from OpenDebateEvidence, alongside GPT-4o TTS speech synthesis and EchoMimic digital human animation, the system significantly outperforms human-authored cases across all metrics in expert evaluations (Quality 4.32 vs 3.65) and achieves an 85% win rate in simulated matchups.
AHAMask: Reliable Task Specification for Large Audio Language Models without Instructions: By applying binary masking (AHAMask) to attention heads within the Transformer backbone of Large Audio Language Models (LALMs), specific acoustic task functionalities can be reliably triggered without textual instructions, while simultaneously revealing the existence of "acoustic functional pathways" inside LALMs.
Aligning Generative Music AI with Human Preferences: Methods and Challenges: A survey/position paper that systematically reviews three research directions of preference alignment in music generation—MusicRL (large-scale RLHF, ~300k preference pairs), DiffRhythm+ (multi-preference DPO for diffusion models), and Text2midi-InferAlign (inference-time tree search, CLAP +29.4%). It provides an in-depth analysis of unique alignment challenges in the music domain (multi-scale temporal coherence, harmonic consistency, cultural subjectivity, and the evaluation paradox) and proposes a future roadmap.
CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation: CCFQA is proposed—the first cross-lingual and cross-modal benchmark covering 8 languages and 14,400 fully parallel speech-text factuality QA samples. It supports four task settings (QA/XQA/SQA/XSQA) and systematically reveals the factual inconsistency of existing MLLMs under language and modality switching. Concurrently, LLM-SQA is introduced, achieving cross-lingual spoken QA transfer via English bridging with only five shots, obtaining an F1 of 51.4 on XSQA, outperforming GPT-4o-mini-Audio (45.7).
Characterizing AI Manipulation Risks in Brazilian YouTube Climate Discourse: This paper analyzes 226k climate change videos and 2.75M comments on Brazilian YouTube using a psycholinguistic framework, revealing that emotional/moral rhetoric significantly drives user engagement. It also demonstrates that fine-tuned LLMs can automatically generate highly engaging climate-denial comments, warning of the potential risks of generative AI in public opinion manipulation.
Cross-Space Synergy: A Unified Framework for Multimodal Emotion Recognition in Conversation: The proposed Cross-Space Synergy (CSS) framework simultaneously addresses the two key challenges of insufficient fusion expressiveness and multi-objective gradient conflicts in multimodal conversational emotion recognition using a two-pronged approach: Synergistic Polynomial Fusion (SPF) in the representation space and a Pareto Gradient Modifier (PGM) in the gradient space.
DeformTrace: A Deformable State Space Model with Relay Tokens for Temporal Forgery Localization: DeformTrace is proposed, introducing deformable dynamic receptive fields and a relay token mechanism into the State Space Model (SSM). By combining the global modeling capabilities of Transformers with the efficient inference of SSMs, it achieves SOTA accuracy and significant efficiency improvements in temporal forgery localization.
Diff-V2M: A Hierarchical Conditional Diffusion Model with Explicit Rhythmic Modeling for Video-to-Music Generation: Diff-V2M is proposed, a hierarchical conditional diffusion Transformer-based video-to-music generation framework. By integrating emotional, semantic, and rhythmic features through explicit rhythmic modeling (low-resolution ODF) and hierarchical cross-attention mechanisms, it achieves SOTA performance on both in-domain and out-of-domain datasets.
DiffA: Large Language Diffusion Models Can Listen and Understand: Proposed DIFFA, the first Large Audio-Language Model based on a diffusion language model. By using a frozen LLaDA-8B backbone, a lightweight dual-adapter architecture, and a two-stage training pipeline, it achieves performance competitive with autoregressive baselines on MMSU, MMAU, and VoiceBench using only 960 hours of ASR data and 127 hours of synthetic instruction data.
Do LLMs Feel? Teaching Emotion Recognition with Prompts, Retrieval, and Curriculum Learning: This work proposes the PRC-Emo framework, which integrates explicit/implicit emotion prompts, a dedicated retrieval repository, and curriculum learning strategies to comprehensively enhance LLM performance on the Emotion Recognition in Conversation (ERC) task, achieving state-of-the-art (SOTA) results on the IEMOCAP and MELD benchmarks.
DualSpeechLM: Towards Unified Speech Understanding and Generation via Dual Speech Token Modeling: This paper proposes the DualSpeechLM framework, which leverages an understanding-driven speech tokenizer (USTokenizer) to extract high-level semantic tokens as LLM inputs and acoustic tokens as outputs. This approach simultaneously optimizes speech understanding and generation capabilities within a unified, end-to-end framework.
End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answering: This paper proposes CLSR, an end-to-end contrastive language-speech retriever. By first converting acoustic representations into text-like representations and then aligning them with text, it efficiently extracts question-relevant segments from long-form audio, providing RAG support for downstream LALM-based long spoken question answering.
Factor(U,T): Controlling Untrusted AI by Monitoring their Plans: This study investigates the safety of the Factor(U,T) protocol, where an untrusted AI decomposes tasks and a trusted AI executes them. It is found that the AUROC for monitoring decomposition plans is only 0.52 (near random), whereas monitoring specific code implementations reaches 0.96. This indicates that malicious intent is difficult to detect in abstract plans but is exposed in concrete implementations, leading to the conclusion that "structural prevention (trusted decomposer) is superior to post-hoc monitoring."
GOMPSNR: Reflourish the Signal-to-Noise Ratio Metric for Audio Generation Tasks: By replacing the instantaneous phase with omnidirectional phase derivatives to reconstruct the SNR metric, GOMPSNR is proposed as a more reliable audio quality evaluation metric, deriving a new family of loss functions that significantly improve neural vocoder performance.
Hearing More with Less: Multi-Modal Retrieval-and-Selection Augmented Conversational LLM-Based ASR: MARS proposes a multi-modal retrieval-and-selection approach to select the most relevant historical context for conversational LLM-ASR (instead of using a fixed set of preceding sentences or the entire history). It outperforms the SOTA system TEA-ASLP, which is trained on 179K hours of data, using only 1.5K hours of training data.
HPSU: A Benchmark for Human-Level Perception in Real-World Spoken Speech Understanding: This work proposes the HPSU benchmark, which includes over 20,000 expert-annotated Chinese and English samples across 16 tasks. It systematically evaluates the deep perception and reasoning capabilities of Speech LLMs in real-world spoken scenarios, revealing a significant gap between the strongest model (Gemini 2.5 Pro, 62.6%) and human performance (87.3%).
HQ-SVC: Towards High-Quality Zero-Shot Singing Voice Conversion in Low-Resource Scenarios: This work proposes the HQ-SVC framework, which jointly extracts content and speaker features based on a decoupled audio codec (FACodec). Combined with an Enhanced Voice Adapter (EVA) to fuse acoustic features such as pitch and energy, progressive optimization is performed using DDSP and a diffusion model. Using a single RTX 3090 and less than 80 hours of singing voice data, the framework achieves zero-shot singing voice conversion quality that outperforms large-scale training baselines, while additionally supporting speech super-resolution tasks.
Improving Multimodal Sentiment Analysis via Modality Optimization and Dynamic Primary Modality Selection: Proposes the MODS framework, which eliminates redundant non-verbal modalities via Graph-based Dynamic Compression (GDC), and designs a sample-level dynamic primary modality selector (MSelector) along with a primary-modality-centric cross-attention (PCCA) mechanism to achieve adaptive dominant modality selection per sample in MSA.
Let the Model Learn to Feel: Mode-Guided Tonality Injection for Symbolic Music Emotion Recognition: Through a MoGE diagnosis strategy, it is systematically discovered that MIDIBERT fails to effectively encode mode-emotion associations. A MoFi injection framework is proposed to inject major/minor mode priors into the first layer of MIDIBERT (the layer with the weakest emotional information identified by diagnosis) via the FiLM mechanism, achieving an accuracy of 75.2% (+11.8%) on EMOPIA and 59.1% (+11.8%) on VGMIDI, with F1 score improvements of 12.3%/15.5%.
Listening Between the Frames: Bridging Temporal Gaps in Large Audio-Language Models: This paper proposes TimeAudio, which endows large audio-language models (LALMs) with precise temporal localization capabilities and end-to-end long audio understanding through three key modules: Temporal Markers, Absolute Time-aware Encoding, and Segment-level Token Merging. It also constructs the FTAR dataset for instruction tuning in fine-grained temporal reasoning.
MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglement: The MF-Speech framework is proposed to disentangle speech signals into high-purity independent content, timbre, and emotion factor representations via multi-objective optimization. It utilizes dynamic fusion and Hierarchical Style Adaptive Normalization (HSAN) to achieve fine-grained, compositional speech generation control, significantly outperforming existing methods on multi-factor compositional speech generation tasks (WER=4.67%, SECS=0.5685).
Multi-granularity Interactive Attention Framework for Residual Hierarchical Pronunciation Assessment: This paper proposes the HIA framework, which achieves bidirectional information interaction among the three granularities of phonemes, words, and sentences via an Interactive Attention Module. Combined with a residual hierarchical structure to alleviate feature forgetting, it achieves SOTA performance across all granularity and aspect metrics on the speechocean762 dataset.
PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment Analysis: Proposes the PaSE framework to explicitly address modality competition in multimodal sentiment analysis through a two-stage optimization strategy involving prototype-guided calibration alignment (Entropic Optimal Transport) and Shapley-value gradient modulation.
PSA-MF: Personality-Sentiment Aligned Multi-Level Fusion for Multimodal Sentiment Analysis: This work introduces a pre-trained personality model into Multimodal Sentiment Analysis (MSA) for the first time to extract personalized sentiment features. Through personality-sentiment contrastive learning alignment and a progressive multi-level (pre-fusion \(\rightarrow\) cross-modal interaction \(\rightarrow\) enhanced fusion) fusion architecture, it achieves SOTA performance on CMU-MOSI and CMU-MOSEI.
REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation: This paper proposes REINA (Regularized Entropy INformation Adaptation), a loss function grounded in mutual information theory that efficiently converts a non-streaming speech translation model into a streaming simultaneous speech translation model. REINA achieves state-of-the-art streaming translation performance across multiple language directions and introduces a new streaming efficiency metric, NoSE.
Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding: VARSTok is proposed, which is the first fully dynamic variable-frame-rate speech tokenizer. Through temporal-aware density peak clustering and implicit duration coding, it achieves adaptive token allocation, outperforming fixed-frame-rate baselines while using fewer tokens.
SpikCommander: A High-Performance Spiking Transformer with Multi-View Learning for Efficient Speech Command Recognition: The authors propose SpikCommander, a fully spike-driven Transformer architecture. By jointly enhancing temporal and channel feature modeling via Multi-View Spike Temporal-Aware Self-Attention (MSTASA) and Spiking Context Refinement MLP (SCR-MLP), SpikCommander outperforms state-of-the-art SNN methods on SHD, SSC, and GSC benchmarks with fewer parameters.
A Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis: This paper proposes the TEXT model, which leverages MLLMs to generate natural language explanations for audio and video to enhance modal representations. It designs a lightweight temporal alignment module combining the merits of Mamba and temporal cross-attention, and employs text-routed sparse mixture-of-experts for cross-modal fusion, comprehensively outperforming SOTAs and large models like GPT-4o on four MSA datasets.
Towards Authentic Movie Dubbing with Retrieve-Augmented Director-Actor Interaction Learning: Authentic-Dubber simulates the interaction between directors and actors in real dubbing workflows. By constructing a multimodal reference footage library, applying an emotion-similarity-based retrieval-augmented strategy, and designing a progressive graph-based speech generation method, it significantly enhances the emotional expressiveness of automatic movie dubbing, achieving SOTA emotional accuracy and MOS scores on the V2C-Animation dataset.