🎵 Audio & Speech¶
💬 EMNLP2026 · 31 paper notes
📌 Same area in other venues: 🧠 NeurIPS2026 (54) · 🎞️ ECCV2026 (24) · 📷 CVPR2026 (23) · 🔬 ICLR2026 (80) · 💬 ACL2026 (70) · 🧪 ICML2026 (36)
🔥 Top topics: Speech & Audio ×24 · Translation ×5 · Dialogue ×3 · Reasoning ×2 · Adversarial Robustness ×2
- A Family Affair: Counterintuitive transfer effects across speech-language models
-
Challenging the long-standing dogma that closely related languages provide the best transfer donor in low-resource speech recognition, this paper demonstrates on seven Formosan endangered languages that two-stage fine-tuning with phylogenetically unrelated donors yields significantly larger WER reductions than related donors, frequently outperforming models trained with doubled in-language data.
- Audio-Reward-R1: Reasoning Reward Model for Unified Audio Understanding and Generation
-
Audio-Reward-R1 is the first unified reasoning-based reward model for both Audio Understanding Preference (AUP) and Audio Generation Preference (AGP) tasks, combining two-stage fine-grained data curation with verifiable-reward GRPO reinforcement learning to achieve state-of-the-art results on Audio-Reward-Bench (87.90% AUP and 80.10% AGP), outperforming Gemini-3-pro-preview.
- Better Retrieval, Worse Robustness: How Multi-hop RAG Amplifies Upstream ASR Errors
-
This paper empirically evaluates spoken multi-hop retrieval-augmented generation (RAG) pipelines across four English accents and three multi-hop QA benchmarks, revealing that while advanced extensions like entity-graph linking and iterative reformulation achieve higher clean-text F1, they amplify upstream ASR errors by expanding the clean-to-corrupted performance gap by 36% to 67%, with query-entity corruption identified as the primary catalyst of catastrophic retrieval breakdown.
- Beyond Text Following: Repairable Arbitration Reversals in Audio-Language Models
-
Addressing the arbitration failure where audio-language models blindly follow conflicting text over clear acoustic evidence, this paper proposes Gated Audio Counterfactual Logit Correction (GACL), a training-free decoding rule grounded in causal activation patching that interpolates joint logits toward a same-audio reference branch to rescue audio decisions under strict faithfulness constraints.
- BiMTokenizer: Preserving Semantic-Acoustic Balance in Low-Bitrate Speech Tokenization
-
BiMTokenizer challenges the recent architectural bloat of dual-tower speech codecs by revitalizing the single-tower paradigm with a bidirectional Mamba backbone and non-parametric Residual Spherical Leech Quantization (RSLQ), delivering state-of-the-art acoustic reconstruction and downstream semantic performance at ~1.1 kbps with only 253M parameters.
- Bulbul: A Dataset for Dialectal Arabic Speech Recognition
-
BULBUL is a community-driven multi-dialect Arabic automatic speech recognition dataset spanning 275 native speakers across 11 Arab countries, providing 71.59 hours of speech covering 16 dialects/sub-dialects alongside accented Modern Standard Arabic and Classical Arabic subsets with two-level human verification.
- CALM: Class-Conditional Sparse Attention Vectors for Large Audio-Language Models
-
CALM introduces a fine-tuning-free classifier that models internal attention heads of frozen large audio-language models as class-conditional experts, aggregating them via margin-weighted soft voting to achieve substantial gains over uniform voting across audio, audiovisual, and visual few-shot benchmarks.
- CSAVocoder: A Causal Spatial Audio Vocoder Towards Real-Time Spatial Audio Generation
-
Addressing the failure of conventional neural vocoders to preserve multi-channel inter-channel acoustic cues and their non-causal streaming latency bottlenecks, CSAVocoder integrates an Attentional Mel Adaptor with dynamic pose FiLM modulation, a strictly causal stateful PixelShuffle generator, and an axial-attention Spatial Consistency Discriminator with format-aware physical losses, delivering superior spatial rendering fidelity alongside sub-real-time streaming efficiency (RTF = 0.1587).
- DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMs
-
Addressing the absence of explicit cross-attention in decoder-only SpeechLLMs, DOA proposes a completely training-free streaming decision policy that slices a proxy cross-attention matrix directly from self-attention weights and pairs it with punctuation-aware history pruning, delivering offline-competitive quality and low latency on continuous long-form audio.
- EmbodiChat: Towards Zero-Latency Text-driven Talking Face Generation for Embodied Dialogue
-
EmbodiChat resolves the trade-off between latency, lip synchronization, and hardware transferability in embodied dialogue by aligning facial animation directly to the intermediate acoustic latent space of two-stage TTS models without cross-modal adapters, delivering a ~150 ms Time-to-First-Audio on a consumer GPU and enabling robust expression transfer across heterogeneous humanoid robots via a Universal Facial Action Compiler (UFAC).
- FlexiSLM: A Spoken Language Model with Dynamic and Controllable Frame Rates
-
FlexiSLM is the first spoken language model featuring dynamic and controllable frame rates across both speech input and output, leveraging adaptive cosine frame merging and explicit sinusoidal conditioning to operate anywhere \(\le 12.5\text{ Hz}\) (including \(6.25\text{ Hz}\), \(5.0\text{ Hz}\), and \(4.0\text{ Hz}\)) without retraining, roughly halving inference time at \(6.25\text{ Hz}\) while outperforming fixed-rate 7B baselines.
- From Visual Cues to Spoken Narration: Rethinking Audio Description Generation
-
This paper re-evaluates automatic Audio Description (AD) by showing that the visual cue window (what to describe) and the spoken narration window (when to voice it) routinely exhibit temporal offsets, introducing Cue2Narrate—a two-stage framework pairing an audio-visual dual-head localizer with a window-conditioned VLM trained via a Description Ranking Loss—and releasing the LongLSMDC long-form benchmark.
- HyperStyler: Low-resource Authorship Style Transfer via Context-aware Style Navigation and Hypernetworks
-
HyperStyler decouples low-resource authorship style transfer into style selection and style realization, using Stylo-navigator to infer context-dependent style coordinates and Stylo-hypernet to dynamically generate decoder prefixes and low-rank adapters, effectively mitigating mode averaging and content-style entanglement.
- KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness
-
KoALa-Bench introduces the first comprehensive benchmark for evaluating Korean speech understanding and speech faithfulness in Large Audio Language Models (LALMs), revealing widespread reliance on textual parametric priors over acoustic evidence in counterfactual and long-form speech contexts.
- LoMaR: Local Margin Preservation via Residual Subspace Adaptation for Noisy Prompt Robustness in Small Language Models
-
Addressing the vulnerability of on-device small language models (SLMs) to speech ASR transcription errors and mobile typing noise caused by narrow logit margins, LoMaR introduces a two-stage PEFT framework that first widens local clean decision boundaries and then restricts noise adaptation to an SVD-isolated residual subspace, significantly reducing token-level rank flips with zero extra inference latency.
- NaijaS2ST: A Multi-Accent Benchmark for Speech-to-Speech Translation in Low-Resource Nigerian Languages
-
Addressing the severe lack of multi-accent parallel speech resources for low-resource African languages, this work introduces NaijaS2ST, a ~269-hour clean parallel speech corpus covering Hausa, Igbo, Yorùbá, Nigerian Pidgin, and English, establishing a rigorous benchmark across cascaded, end-to-end, and AudioLLM paradigms in bidirectional S2TT and S2ST tasks.
- Not All Queries Are Equal: Query-Type-Aware Mixture-of-Experts for Temporal Knowledge Graph Forecasting
-
Addressing the limitation of conventional temporal knowledge graph forecasting models that apply a monolithic reasoning pipeline to all queries regardless of historical evidence patterns, QTI-MoE categorizes queries into sparse, recurrent, and ambiguous regimes and deploys specialized experts equipped with complementary inductive biases (recency decay, stable count accumulation, and diversity normalization) alongside an offline-distilled query-aware router, achieving state-of-the-art results across four major benchmarks.
- OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation
-
OmniRetriever addresses the persistent neglect of fused multimodal signals during the training of unified AVT encoders by using a stop-gradient copy of the joint embedding as a distillation teacher for single-modal sub-encoders, combined with modality-cycled Tuple-InfoNCE to surpass Gemini Embedding 2 across a 12-direction retrieval benchmark.
- Overtone: Fine-Grained Audio Reasoning via Diagnostic-Driven Delayed Acoustic Fusion
-
Addressing the "representation reorganization" bottleneck in audio LLMs—where middle layers systematically overwrite fine-grained acoustics into semantic abstractions—Overtone introduces a diagnostic-driven delayed acoustic fusion architecture that extracts dual semantic/acoustic tracks from model-localized segments and injects frozen WavLM features into post-compression layers (L12–L24) via gated cross-attention with reconstruction regularization, achieving state-of-the-art results on MMAU among open-source \(\le\)8B models (77.66%).
- ParaBridge: Bridging Paralinguistic Perception and Dialogue Behavior in Speech Language Models
-
ParaBridge introduces an on-policy self-distillation framework that transfers dense, full-vocabulary next-token probability targets from an inference-time paralinguistic prompt scaffold into a scaffold-free student policy, bridging the perception-behavior gap in speech language models without requiring curated dialogues, human labels, or external reward models.
- Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation
-
This paper frames TTS-friendly text generation as a multi-objective preference alignment problem, leveraging the interpretable Feature-aware Sampling and Tuning (FaST) framework and a comprehensive evaluation suite to make LLMs natively output speakable text without post-processing latency, excelling even under extreme 10-sample data constraints.
- SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision
-
SimulS2ST-Omni introduces a joint text-code trajectory supervision mechanism paired with a decoupled Thinker–Talker architecture, achieving state-of-the-art long-form streaming speech-to-speech translation using only ~2k hours of synthesized paired audio.
- SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision
-
SimulS2ST-Omni establishes an explicit joint text-code trajectory supervision framework paired with a decoupled Thinker–Talker architecture, delivering state-of-the-art sentence-level and long-form streaming S2ST using only ~2,000 hours of paired cross-lingual audio.
- Speaking While Listening: A Survey and Empirical Audit of Full-Duplex Spoken Dialogue Systems
-
This survey and empirical audit introduces three complementary frameworks—an L0–L3 Architectural Hierarchy, a \(T \times I \times R\) Interaction Ontology, and a 5-state Decision State Machine—to dismantle the ambiguity surrounding "full-duplex" spoken dialogue systems, proving through independent Full-Duplex-Bench v1.5 evaluations that observed performance is fundamentally constrained by training data cell coverage rather than architectural reach alone.
- SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing
-
SpeechEditBench establishes the first bilingual, multi-attribute benchmark for instruction-guided speech editing covering seven atomic dimensions and multi-operation compositions, introducing an anchor-based evaluation protocol (target, preservation, and joint success) to systematically diagnose capability fragmentation and content preservation bottlenecks in modern Speech LLMs.
- Tracing a Sparse Emotion-Control Circuit in LLM-Based Text-to-Speech
-
This paper introduces a codec trajectory score and a late residual direction score to measure emotion shifts in LLM-based TTS, localizes a sparse source-to-readout emotion-control circuit comprising 23–27 attention heads and MLPs (~5% of all components) that recovers/suppresses 74–88% of the readout shift, and demonstrates that this internal circuit causally drives pitch, energy, and spectral changes in decoded audio far more effectively than global residual steering.
- TTS-Hub: Leveraging Modular LoRAs and Arithmetic Composition for Controllable Text-to-Speech
-
TTS-Hub is the first framework that formulates controllable text-to-speech synthesis as an algebraic arithmetic composition of modular LoRA adapters, constructing a repository of 32 fine-grained attribute specialists across 6 categories to enable fine-grained, plug-and-play multi-attribute speech control without retraining the frozen backbone.
- Unlocking the Spoken Style: Mechanistic Localization and Lightweight Control of Orality in LLMs
-
Addressing the issue where voice agents produce overly written responses and lack audience-adaptive intensity control, this paper constructs WSPairs—the first parallel corpus of 4,709 written-to-spoken pairs annotated with 0–5 Orality Levels—mechanistically localizes orality to late Transformer layers and sparse attention heads via causal patching, and proposes two lightweight controllers: weight-free Calibrated Activation Steering (CAS) and Learned Activation Steering (LAS) requiring only 0.004%–0.006% trainable parameters to achieve fine-grained, monotonic style rewriting.
- Unsupervised Speech Recognition at the Syllable Level
-
SylCipher introduces the first syllable-level unsupervised speech recognition system that jointly segments continuous speech into discrete syllable tokens and aligns them with unpaired text via information-constrained masked language modeling and positional marginal matching, completely bypassing G2P phoneme dictionaries and GAN instability while outperforming prior G2P-free systems by up to 40% relative CER.
- When Scaling Fails: Mitigating Audio Perception Decay of LALMs via Multi-Step Perception
-
Addressing the counter-intuitive phenomenon where extended Chain-of-Thought reasoning causes Large Audio-Language Models to lose acoustic grounding and suffer from "Audio Perception Decay," this paper introduces the CAFE evaluation framework to quantify reasoning-time audio fidelity and proposes MPAR², a multi-step perception and review strategy powered by GRPO that achieves 74.59% on MMAU while adaptively scaling inference compute.
- Why We Need Speech to Evaluate Speech Translation
-
This paper benchmarks text- and speech-aware quality estimation (QE) metrics on speech-specific phenomena, trains the SpeechCOMET family and evaluates SpeechLLMs as judges, revealing that existing metrics perform at chance level on gender and prosodic distinctions despite speech input.