🎵 Audio & Speech¶
🧠 NeurIPS2026 · 54 paper notes
📌 Same area in other venues: 🎞️ ECCV2026 (24) · 📷 CVPR2026 (23) · 🔬 ICLR2026 (80) · 💬 ACL2026 (70) · 🧪 ICML2026 (36) · 🤖 AAAI2026 (30)
🔥 Top topics: Speech & Audio ×37 · Dialogue ×5 · Compression ×3 · Adversarial Robustness ×3 · LLM ×3
- Adaptive Compression and Targeted Perturbation: A Unified Framework for Generalized Audio Deepfake Detection
-
To tackle heterogeneous spoofing artifact densities and the pathological robustness-accuracy trade-off in audio deepfake detection, this paper introduces a unified Whisper-based framework integrating an adaptive multi-resolution compression pool (AAFI) and targeted high-frequency mel-band adversarial perturbation (Mel-BAP), achieving state-of-the-art zero-shot cross-domain and cross-lingual detection.
- AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency
-
AnyEdit decouples semantic editing from acoustic rendering within a separation-free two-stage framework, combining a noise-robust prosody tokenizer, an autoregressive editor, and an in-context flow-matching renderer to achieve high-fidelity speech and singing editing under real-world noise, reverberation, and background music.
- Asymmetric Hierarchical Anchoring for Robust Audio–Visual Cross-Modal Generalization
-
Addressing the information allocation ambiguity and semantic leakage inherent in symmetric audio-visual frameworks, AHA introduces an asymmetric architecture anchored on the audio RVQ hierarchy, coupled with GRL-based adversarial decoupling and local sliding alignment, to significantly boost zero-shot cross-modal event localization.
- Audible World Models: Spatially Aware Sound Generation for 3D Worlds
-
Audible World Models connects panorama generation, semantic source parsing, dry-audio synthesis, and geometric acoustics to store sound as persistent 3D world state; it produces more accurate motion-dependent directional cues while retaining strong semantic alignment, but currently remains an offline construction system for static worlds.
- Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos
-
Addressing the short-clip bias and deep-layer visual dominance of existing multimodal LLMs, AV-Flamingo introduces a fully open-source audio-visual foundation model trained on the 7M-instance AV-Skills dataset with a three-stage curriculum and timestamp-grounded Temporal Audio-Visual Interleaved CoT (TAVIT), matching frontier proprietary models across 15+ long-form omnimodal benchmarks.
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation
-
AudioCALM establishes the Continuous Autoregressive Language Modeling (CALM) framework by replacing discrete token prediction with a rectified-flow head over continuous audio latents, coupled with block-causal AR-Flow attention and Asymmetric Mixture-of-Modality-Experts (A-MoME) to achieve state-of-the-art unified speech, sound, and music synthesis.
- Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation
-
Baton introduces explicit semantic planning to joint video-audio generation by utilizing an MLLM planner (VA-Planner) to synthesize mutually aligned, keyframe-level perceptual blueprints prior to denoising, combined with Relative Semantic RoPE to bridge asymmetric spatio-temporal grids and prevent cross-modal trajectory drift.
- Breaking Adversarial Transferability in Fine-Tuned Speech Recognition
-
Addressing the vulnerability where fine-tuned private ASR systems are compromised by adversarial perturbations crafted on open foundation models, TransferBreaker combines base-model adversarial fine-tuning, latent Jacobian regularization, and hybrid-gradient adversarial training to slash transfer Word Error Rate (WER) from 92.6% down to 27.8% while preserving clean recognition accuracy.
- CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognition
-
Introduces CS-Dialogue, the first freely accessible 104-hour large-scale speech dataset of spontaneous Mandarin-English code-switching dialogues with complete transcriptions from 200 speakers, and establishes comprehensive ASR benchmarks across scratch-trained architectures and state-of-the-art foundation models.
- CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognition
-
To address the scarcity of natural, full-length conversational speech resources in code-switching, this paper presents CS-Dialogue, a 104-hour spontaneous Mandarin-English dialogue dataset featuring 200 speakers with comprehensive transcriptions and multi-paradigm ASR benchmarking.
- Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning
-
To resolve the performance trade-off between interactive conversational dynamics and complex reasoning in full-duplex spoken dialogue models, this paper introduces DuplexPO, a reinforcement learning framework that decouples real-time floor-control decisions ("when to speak") from semantic content generation ("what to say") via window-level policy optimization.
- DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
-
DELTA-TTS introduces the first parameter-efficient adaptation framework that converts a pretrained autoregressive (AR) TTS model into a discrete diffusion language model (dLLM), training only ~15% added parameters (LoRA + Conformer convolution) on 585 hours of LibriTTS to surpass its CosyVoice3 backbone in WER while speeding up token generation by 3.3×.
- Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling
-
Challenging the prevailing reliance on large generative language models and brittle discrete phoneme pipelines in speech brain-computer interfaces, this paper proposes BEST, a lightweight two-stage framework that integrates hierarchical neural representations via multi-scale continuous convolutional fusion and maps them to text using a compact speech-aligned decoder, achieving word error rates of 6.19% on Brain-to-Text '24 and 1.62% on '25 with merely 3.85% of comparable model parameters.
- DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech
-
DUET identifies linearly decodable emotion directions in frozen hidden states that are nearly orthogonal to speaker identity, and introduces a plug-and-play dual-space control framework combining representation steering and mel-space differentiable vocoder guidance to outperform 10 supervised emotional TTS baselines across five backbones and three benchmarks.
- EchoChat: Structured Cognitive Reasoning in Empathetic Spoken Dialogue
-
EchoChat reformulates empathetic spoken dialogue from direct input-to-response generation into a structured cognitive reasoning process spanning perception, Theory-of-Mind inference, and emotionally aligned response generation, backed by Acoustic-Anchored Attention (AAA) in SFT and Step-Decomposed Credit Assignment (SDCA) in RL to resolve empathetic mimicry and cascaded error propagation.
- Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes
-
This paper introduces RealSSA—a realistic benchmark with dynamic multi-speaker interference, reverberation, sudden noise, and a real-recorded evaluation set—along with Falcon, an ultra-lightweight single-pass causal audio-visual separator trained with an encoder-space speaker contrastive loss that matches or beats heavy baselines while slashing GPU latency by 11.1×.
- Enhancing Speech Large Language Models through Reinforced Behavior Alignment
-
VIRBA introduces a reinforcement learning framework that structures multi-view spoken instruction groups across speakers, accents, noise, and disfluencies, combining verifiable, invariant, and adaptive reasoning rewards optimized via Cross-Acoustic Group Relative Policy Optimization (CA-GRPO) to bridge the SpeechLM reasoning and robustness gap.
- FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries
-
FLARE is the first full-modality long-video audiovisual retrieval benchmark, featuring 87,697 finely segmented clips and 274,933 user-style queries filtered by a hard bimodal constraint to expose the performance drop under short user queries and the bottleneck in audio-language alignment.
- From Isolated feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment
-
This paper introduces an unsupervised multi-view Sparse Autoencoder (SAE) alignment framework grounded in pitch transposition equivariance, which recovers ordered relational orbit structures (e.g., chord rings, subdominant key rings, and melodic chains) within music foundation models and enables full concept family grounding from minimal anchor examples.
- Getting Motif-ated: Controllable AI Compositions from Injected Motif Prompts
-
MotiGen enables fine-grained thematic control in pretrained symbolic music language models by conditioning on transposition-invariant interval-class contours, reinforcing prompt saliency through matched scalar attention bias, and learning the association via a two-phase curriculum from focused phrase crops to full musical scores.
- GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding
-
Departing from the conventional paradigm of codebook- or grid-partitioned latent spaces, GS-Codec introduces a 1D Gaussian Splatting bottleneck for neural audio coding, pairing parametric continuous basis decomposition with a feed-forward predictor net to achieve single-pass inference and fine-grained, training-free post-quantization bitrate control.
- Hearing is Believing? Evaluating and Analyzing Audio Language Model Sycophancy with SYAUDIO
-
This paper introduces SYAUDIO, the first dedicated benchmark to evaluate sycophancy in Audio Language Models (ALMs), discovering that spoken user instructions trigger an audio-specific sycophancy avalanche, and proposes mitigation via SFT and decode-time hidden-state steering.
- Helping Music Co-Creation Agents 'Listen' Well: Hierarchical Self-Supervised World Models for Understanding and Generation
-
The paper introduces a hierarchical self-supervised world model and conditional flow-matching framework for symbolic music (MIDI), leveraging a Swin V2 encoder to align representation levels with musical time scales and enabling sub-3-second CPU interactive inpainting and co-creative suggestion generation.
- Kinetic-Optimal Scheduling with Moment Correction for Metric-Induced Discrete Flow Matching in Zero-Shot Text-to-Speech
-
Addressing the reliance on heuristic temperature schedulers and finite-step CTMC path-tracking errors in metric-induced discrete flow matching (MI-DFM), this paper proposes a training-free kinetic-optimal scheduler traversing probability paths at constant Fisher–Rao speed and a lightweight finite-step moment correction, establishing GibbsTTS which achieves state-of-the-art naturalness and speaker similarity in zero-shot codec-based TTS.
- LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale
-
LibriBrain100 introduces an open-source, 104-hour non-invasive MEG continuous speech decoding benchmark combining an 80.5-hour deep single-subject repository with a 32-subject broad cohort, demonstrating that supervised fine-tuning of foundation models on deep data effectively transfers to data-constrained new subjects on word classification.
- MAEB: Massive Audio Embedding Benchmark
-
The Massive Audio Embedding Benchmark (MAEB) establishes a unified, compute-efficient evaluation standard across 30 tasks in 100+ languages, evaluating 53 foundation encoders to uncover fundamental acoustic-linguistic trade-offs, pervasive clustering failures, and predictive alignment with downstream Audio LLMs.
- MAST: Label-Efficient, Robust, and Generalizable Sound Detection for Biodiversity Monitoring via Masked Audio Pretraining and Self-Training
-
Addressing the severe scarcity of time-frequency annotations and catastrophic performance collapses under distribution shift in bioacoustic monitoring, MAST integrates masked audio spectrogram reconstruction, an asymmetric upsampling adapter, and a confidence-weighted two-stage self-training curriculum to achieve highly robust sound localization using only a single day of single-site labels.
- Mind The Gap (MTG-AD): Dataset and Fine-grained Evaluation for End-to-End Inline Audio Descriptions
-
MTG-AD establishes the first 36-hour benchmark of untrimmed user-generated videos co-authored by paired blind and sighted professionals for inline audio description, alongside MTG-AD-Eval, an end-to-end reference-free evaluation suite integrating TTS speech rendering with multimodal LLM judges that significantly correlates with sighted and blind human judgments.
- Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
-
TextPro-SLM tackles the persistent modality gap in speech large language models from the input side by decoupling spoken input into synchronized text tokens and reconstruction-supervised compact prosody embeddings, reducing the modality gap of a 7B model to 0.7% with only roughly 1,000 hours of audio training.
- Multi-Party Backchannel Prediction: a Diagnosis, a Benchmark, and a Ceiling
-
This paper establishes the first person-disjoint continuous backchannel prediction benchmark for multi-party natural meetings, proves that zero-shot dyadic model failure stems from the temporal predictor rather than acoustic representations, and reveals an inherent cross-listener generalization ceiling driven by idiosyncratic listener behavior that resists capacity reduction, adversarial identity removal, and oracle lexical conditioning.
- MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation
-
MultiTalk introduces an automated data engine generating a 57.6k-hour long multi-party bilingual synthetic corpus alongside MultiTalkBench based on real human meetings, scaling the dual-stream Moshi architecture to coherent, multi-speaker, English-Chinese full-duplex conversation without structural modifications.
- Native Audio-Visual Alignment for Generation
-
Addressing the trade-offs of late-stage cross-modal alignment in dual-tower designs and semantic-synchronization entanglement in unified tri-modal models, NAVA proposes a native audio-visual alignment framework with an Align-then-Fuse MMDiT and Timbre-in-Context Conditioning, achieving state-of-the-art synchronization, visual quality, and multi-speaker timbre control with only 6.3B parameters.
- OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning
-
OmniCapBench reframes omnimodal audio-visual captioning evaluation from free-form text into atomic, verifiable units across references, events, and shots, combining deterministic rule checks with bounded local LLM judges to eliminate scoring instability and uncover deep-seated perceptual bottlenecks.
- OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations
-
OpenWhistle organizes years of underwater recordings from one dolphin group into a pretraining corpus containing approximately 180,000 whistles and an expert-annotated subset of 8,354 whistles, establishing session-split classification and detection benchmarks with frozen-representation linear probes; corpus-pretrained Wav2Vec2.0 achieves 81.1% classification accuracy and 75.6% detection mAP, without decoding the meaning of dolphin communication.
- Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering
-
Addressing the "state inertia" failure mode where full-duplex spoken language models miss the opening words of user interruptions due to internal representations lagging in a generative state, this paper reveals the dynamic modulation between generative and perceptive states via Logit Lens, introduces the Zero-Buffer Benchmark (ZBB) to diagnose initial-word semantic loss, and proposes a training-free activation steering method using a perception vector that substantially restores interruption comprehension.
- Passing: An Endless Journey through Reconstructed Spacetime with AI-Generated Sound
-
Passing resamples a monorail-window recording into a nonlinear journey with branching playback, lets viewer presence indirectly change the visual path, and uses an installation-adapted SpecMaskFoley to generate sound in real time, presenting an exhibition case study and creative observations rather than a controlled experiment in acoustic reconstruction accuracy.
- PitchBench: Measuring Pitch Hearing in Audio-Language Models
-
PitchBench introduces a hierarchical diagnostic benchmark spanning 28 experiments across 3 cognitive levels to systematically evaluate pitch hearing in audio-language models, revealing critical perceptual failures across frontier ALMs, complete breakdown on polyphonic voices, and severe performance inflation under multiple-choice evaluation.
- Predictive–Generative Drift Decomposition for Speech Enhancement and Separation
-
SIPS is a plug-and-play framework based on stochastic interpolants that bridges predictive and generative speech restoration by decomposing sampling dynamics into a predictor-induced deterministic drift and an unconditional clean-speech score prior, achieving high perceptual quality within only 15 steps.
- RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark
-
Grounded in the psychometric Cattell–Horn–Carroll (CHC) theory of human intelligence, RAIL formulates auditory cognition into 5 core dimensions and 32 subcapabilities across 5,306 samples, revealing that current LALMs rely heavily on textual priors while suffering from acute deficits in acoustic perception, non-speech memory, and processing efficiency.
- Reconstructing the Vocal Tract with Differentiable Acoustic Simulation
-
The paper models the vocal tract as a differentiable dynamic acoustic tube and combines frequency-domain solving, smooth turbulence gating, and neural geometry parameterization to infer area functions and MRI visualizations from speech; its default forward synthesis runs at 71.5 times real time, but the reconstructed geometry is an acoustically compatible explanation, not unique anatomical ground truth.
- Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement
-
Addressing long-overlooked bottlenecks in universal speech enhancement (USE), this work demonstrates that time-shifted anechoic clean speech is a superior dereverberation target over conventional early-reflection targets, presents a principled two-stage framework combining frozen regression with residual GAN refinement to optimize the distortion-perception tradeoff, and breaks the performance ceiling through VQScore data filtering and high-quality fine-tuning.
- Rubato: Transcribing Piano Music with Timestamps
-
Rubato introduces a prompt-conditioned encoder-decoder model and the canonical InterMo text representation to transcribe piano audio directly into human-readable sheet music aligned with physical acoustic timestamps, outperforming cascaded pipelines and oracle intermediate representations in notational accuracy.
- SENSE: Semantic Neural Speech Synthesis from Brain Dynamics via Spatial Graph Encoding
-
SENSE combines electrode-geometry graph encoding with EEG semantic conditioning learned only from congruent trials to improve waveform reconstruction on the passive-listening N400 dataset; unseen-subject WER falls from FE-Phoneme's 1.1231 to 0.9948, but remains far from reliable language recovery.
- Structural and Representational Agency in LLM-Driven Timbre Generation and Interpolation
-
Through an acoustic measurement study of Timbre Canvas—a working LLM-driven timbre synthesis and interpolation interface—this paper reveals a profound dissociation between user "effective agency" (reachable audio space is structurally bounded as a zonotope) and "epistemic agency" (displayed perceptual coordinates err by three times the reachable extent, and weaker models fail silently while reporting convergence).
- TAC: Timestamped Audio Captioning
-
Addressing the lack of fine-grained temporal grounding and frequent hallucinations in large audio-language models under polyphonic acoustic scenes, TAC synthesizes multi-granular dynamic mixtures and optimizes a timestamped audio captioner with weighted temporal and speaker losses, serving as an effective semantic bridge for multimodal reasoning.
- TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios
-
TELEVAL introduces a large-scale benchmark of over 40,000 instruction-free audio samples designed to bridge the gap between acoustic perception and conversational action in Chinese Spoken Language Models, establishing decoupled assessments for content fulfillment and interactional appropriateness while exposing the pervasive "Caption Trap."
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models
-
Addressing the collapse of multichannel audio into monaural semantic streams in large audio-language models, this paper formalizes a three-level Auditory Scene Analysis (ASA) task and introduces TWNM, combining First-Order Ambisonics physical simulation, slot-regularized spatial encoding, dense hybrid projection, and progressive preference optimization to enable multi-source attribute binding and spatial cognitive reasoning.
- Transcoda: End-to-End Zero-Shot Optical Music Recognition via Data-Centric Synthetic Training
-
Addressing real-world data scarcity and non-unique symbolic encodings in Optical Music Recognition (OMR), Transcoda develops a 21-stage target normalization engine and realistic degradation pipeline, training a compact 59M model in 6 hours on a single GPU that significantly outperforms billion-parameter models on zero-shot historical scans.
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities
-
UniFlow-Audio proposes the first flow-matching-based non-autoregressive universal audio generation framework that simultaneously unifies both time-aligned and non-time-aligned tasks across text, audio, and visual modalities using a disentangled dual-fusion mechanism.
- V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation
-
V2M-Zero is the first zero-pair video-to-music generation framework that decouples temporal synchronization from semantic musical style via standardized intra-modal event curves, achieving state-of-the-art temporal alignment and audio fidelity without requiring any paired video-music training data.
- VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing
-
VocalCoachBench introduces the first expert-annotated benchmark for evaluating audio-language models on singing feedback, disentangling evaluation into deterministic structured tasks and atomic claim-based diagnosis and correction assessment to reveal critical gaps in fine-grained vocal issue recognition.
- VocalGrad: Evaluating Acoustic Perception in Audio Language Models
-
VocalGrad introduces a controlled 21,600-sample benchmark isolating low-level acoustic perception from high-level reasoning via binary change-direction classification, revealing near-chance (~50%) accuracy across state-of-the-art audio language models and uncovering two distinct representation failure modes through layer-wise linear probing.
- Words That Make Language Models Perceive
-
By appending lightweight sensory instruction cues (e.g., "see" or "hear") and pooling hidden states across autoregressive generation trajectories, purely text-trained LLMs dynamically steer their internal representational geometry toward frozen specialist vision (DINOv2) and audio (BEATs) encoders without any multimodal fine-tuning.
- X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation
-
X-AVDD tackles memory bottlenecks and cross-modal semantic misalignment in audio-visual dataset distillation by introducing a decoupled distillation framework with bidirectional cross-attention, cutting end-to-end synthesis compute by ~8x while drastically outperforming prior SOTA on benchmarks like VGGSound.