ECCV2026 Audio & Speech AI paper notes paper summaries Speech & Audio Multimodal/VLM Personalized Generation Sentiment Analysis Diffusion Models Alignment/RLHF

🎵 Audio & Speech¶

🎞️ ECCV2026 · 23 paper notes

📌 Same area in other venues: 📷 CVPR2026 (22) · 🔬 ICLR2026 (80) · 💬 ACL2026 (71) · 🧪 ICML2026 (36) · 🤖 AAAI2026 (30) · 🧠 NeurIPS2025 (47)

🔥 Top topics: Speech & Audio ×10 · Multimodal/VLM ×4

MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech: MGM-Omni separates multimodal understanding and speech synthesis into brain-mouth tracks, combining dual audio encoders with chunk-based parallel decoding to achieve 4.98% English WER and 0.19 RTF with a 2B SpeechLM on its Long-TTS-Eval benchmark while conditioning generation on a reference voice.
JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching: JAM-Flow treats LivePortrait's low-dimensional mouth expression keypoints and F5-TTS's mel-spectrograms as two mutually maskable modalities inside a single flow-matching framework, trained jointly through a dual-stream MM-DiT with half-fused joint attention, RoPE temporal alignment, and modality-asymmetric attention masks; one ~500M model covers talking head generation, TTS, motion-to-audio, and automated dubbing, producing a 20-second video in 45 seconds on a single RTX A6000.
Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities: PRLF decomposes "which modality is trustworthy right now" into two complementary signals — classification confidence and Fisher information — to pick a per-sample dominant modality, then lets the auxiliary modalities self-refine first and align to the dominant one through projection and residual denoising over four iterations, reaching average F1 of 77.02 / 76.24 / 81.19 on CMU-MOSI / CMU-MOSEI / SIMS under inter- and intra-modality missingness.
CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing: CoSyncDiT organizes voice-style learning, lip-rhythm calibration, and text alignment sequentially inside a flow-matching Transformer, then regularizes alignment with frame-level contrastive learning and CTC, achieving 47.24% speaker similarity, 7.53% WER, and 0.319 Sync-KL on CinePile-Dub zero-shot Setting 2 (original Table 5).
Music-to-Dance Generation via Atomic Movements: The paper models dance explicitly as a sequence of semantically readable, reusable "atomic movements": a discrete diffusion model first plans the type, duration, and temporal position of every atomic movement conditioned on the full music (a symbolic dance score), and a transition-aware continuous diffusion generator then fills in the transitions, retrieves instances by duration-nearest matching and re-creates them under masked noise, achieving the lowest FID (25.26 / 9.03), the highest BAS (0.2470), and the highest structural consistency R (26.6) on AIST++.
Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance: Addressing the limitation where existing video-to-audio (V2A) models can only generate a complete audio track all at once instead of adding sounds layer-by-layer like Foley artists, this paper proposes Negative Audio Guidance (NAG). By training a branch conditioned on "already-generated audio tracks" and utilizing it inversely during sampling, the current generation is pushed away from existing sounds. This enables step-by-step generation of complementary audio tracks to mix into high-quality synthesized sound, using only standard single-reference audio-visual datasets.
HumanOmni-Speaker: Identifying Who said What and When: To tackle the fundamental failure of Omni-modal LLMs in multi-person dynamic conversations ("who said what and when") and their over-reliance on static visual shortcuts, this paper introduces the Visual-Registered Speaker Diarization and Recognition (VR-SDR) benchmark and proposes HumanOmni-Speaker, featuring a 25 fps Visual Delta Encoder that compresses inter-frame motion residuals into just 6 tokens per frame for robust cross-modal spatio-temporal alignment.
JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing: JoVA introduces a streamlined, unified diffusion framework for joint video-audio generation and editing, replacing fragmented cross-modal attention layers with native joint self-attention and channel-wise conditioning alongside an adaptive mouth-area flow matching loss to achieve state-of-the-art multimodal synchronization and instruction-guided editing.
MG-RWKV: Multi-Grained Context-Aware RWKV for Temporal Forgery Localization: MG-RWKV proposes a temporal forgery localization (TFL) framework based on the linear-complexity recurrent architecture of RWKV. By capturing global temporal context via bidirectional RWKV, adaptively selecting explicit temporal receptive fields through a Multi-Grained Mixture of Experts (MG-MoE), and eliminating contradictory predictions across multi-scale features using a Cross-Grained Consistency constraint (CGC), it comprehensively outperforms previous state-of-the-art methods on four benchmarks: Lav-DF, TVIL, Psynd, and AV-Deepfake1M, while maintaining \(O(T)\) linear complexity with an inference time of only 73.4ms.
MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning: MJEPA introduces an elegant and scalable audio-visual self-supervised learning framework that trains a single, unified ViT encoder using only joint-embedding predictive objectives in latent space; by combining intra-modal multi-level feature prediction with cross-modal bidirectional alignment, it sets new state-of-the-art records across seven frozen evaluation benchmarks without requiring contrastive objectives, pixel/spectrogram reconstruction, or heavy data augmentations.
Moving Beyond More Views: Redundancy-Aware Ego–Exo Fusion for Proficiency Estimation: Addressing the performance degradation caused by redundant perspectives and cross-view overfitting when adding exocentric views in Ego–Exo proficiency estimation, this paper introduces AdaMVS to dynamically select the most informative Top-K views at the data level and VIB-GB to balance gradient dynamics and compress latent redundancy at the feature level.
MuSViT: A Foundation Vision Model for Sheet Music Representation: Addressing the failure of general-purpose vision models to capture structured musical notation, MuSViT introduces the first foundation vision transformer for sheet music, pre-trained on 9.7M IMSLP pages via a two-stage MAE curriculum and decisively outperforming general vision backbones and task-specific SOTA across four downstream tasks.
Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space: Addressing the excessive training overhead and reliance on external alignment networks in video-to-audio (V2A) synthesis, this paper proposes Flowley, a single-stage flow-matching framework incorporating Progressive Soft-masked Cross-Attention (PSCA) for parameter-free temporal synchronization, coupled with a sound-aware captioner (SoundCap) to deliver state-of-the-art generation fidelity.
Q-TriM: Question-Guided Tri-Modal Attention for Audio–Visual Question Answering: Addressing the information loss and error propagation caused by deeply stacked attention layers in audio-visual question answering, Q-TriM introduces a shallow, parallel fusion architecture featuring question-guided Tri-Modal Attention (where Query, Key, and Value come from distinct modalities) and adaptive modality mixture, achieving a state-of-the-art 70.89% overall accuracy on MUSIC-AVQA-R.
Sparsity-Inducing Divergence Losses for Biometric Verification: Q-Margin shifts the margin penalty from geometric logit modifications to a probabilistic reference measure in \(\alpha\)-divergence losses. While maintaining posterior sparsity, it consistently outperforms ArcFace/CosFace baselines in low-FAR scenarios on IJB-B/C and VoxCeleb, and reduces training throughput overhead from 27% to 5% via exact top-K truncation.
StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Description: StoryTeller is a training-free framework for long-form audio description that maintains an evolving identity graph and a salience-weighted narrative memory, generating grounded, cross-scene coherent descriptions from raw video and movie titles without task-specific tuning or curated character banks.
Structured-Noise Masked Modeling for Video, Audio and Beyond: This paper introduces a data-independent structured-noise masking framework that generates spatiotemporally smooth 3D Green noise masks for video and regularized blue noise (R-BN) masks with controlled time-frequency patch separation for audio spectrograms, consistently outperforming random masking and expensive data-adaptive strategies with zero additional computational overhead.
C3ASD: Multi-Level Consistency-Driven Representation Learning for Robust Active Speaker Detection: Addressing the severe performance degradation of active speaker detection models under real-world noise and occlusion, C3ASD regularizes representation learning without requiring corrupted training data or architectural bloat via three complementary constraints—embedding-level cross-modal alignment, sequence-level intra-modal contrastive margin enlargement, and prediction-level confidence-masked distillation—achieving superior robustness with merely 1.02M parameters.
HiChor: Hierarchical Choreography Generation from Pop Music with Choreographic Primitives: Addressing the lack of structured artistic choreography in existing music-driven dance generation models, HiChor draws upon foundational choreographic primitives (phrase-level organization, multi-tier beat and energy alignment, and semantic stylization) to establish a hierarchical diffusion framework using training-free ChoreoEdit and LLM-based prompt decomposition, generating long-horizon pop dances with superior phrase diversity, musical alignment, and seamless transitions.
LipsFlow: The First Exploration of Neuromorphic OT-CFM in Multi-Speaker Visual Speech Recognition: LipsFlow is the first to introduce neuromorphic event camera perception and Optimal Transport Conditional Flow Matching (OT-CFM) to multi-speaker visual speech recognition. It captures millisecond-level lip dynamics via a learnable event representation, achieves 2-step deterministic ODE decoding via OT-CFM, and resolves homophenous ambiguities using dual-level semantic supervision, establishing a new SOTA on DVS-Lip with a 22.3% WER and 240ms latency.
Meric: A Unified Framework for Multimodal Music Generation and Retrieval via Representation Space Anchoring: Meric introduces a decoupled architecture anchored on pretrained music semantic representations, training a Flow-DiT acoustic decoder on massive unpaired music and a lightweight diffusion-based Music Head on multimodal pairs, thereby unifying high-fidelity cross-modal music generation and zero-shot retrieval within a single forward pass.
MoMCE: Mixture of Modality and Cue Experts for Multimodal Deception Detection: MoMCE tackles cross-modal conflict and cue heterogeneity in multimodal deception detection through a hierarchical prompt-aware Mixture of Experts architecture coupled with consistency-aware routing entropy regularization and cue diversity constraints, achieving state-of-the-art performance across DOLOS, Bag-of-Lies, and MU3D.
See & Sniff: Learning Visuo-Olfactory Representations: Using the key insight that "odor identity remains invariant to visual transformations within semantic categories," this paper extends the smell-only SmellNet dataset into a paired visuo-olfactory dataset, SmellNet-V. It designs a self-supervised dual-stream framework named See & Sniff based on dense local contrastive alignment. The proposed method significantly outperforms smell-only baselines in odor classification, cross-modal retrieval, and a newly introduced pixel-level odor localization task.