HumanOmni-Speaker: Identifying Who said What and When¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/HumanMLLM/HumanOmni-Speaker
Area: Audio & Speech
Keywords: Multimodal Large Language Models / Audio-Visual Speaker Diarization / Visual Speech Recognition / Spatio-Temporal Alignment / Visual Delta Encoder
TL;DR¶
To tackle the fundamental failure of Omni-modal LLMs in multi-person dynamic conversations ("who said what and when") and their over-reliance on static visual shortcuts, this paper introduces the Visual-Registered Speaker Diarization and Recognition (VR-SDR) benchmark and proposes HumanOmni-Speaker, featuring a 25 fps Visual Delta Encoder that compresses inter-frame motion residuals into just 6 tokens per frame for robust cross-modal spatio-temporal alignment.
Background & Motivation¶
Recent breakthroughs in Omni-modal Large Language Models have demonstrated impressive competence in unifying audio, visual, and textual modalities into a single semantic representation. However, when deployed in complex real-world multi-person conversational environments, these models encounter a critical roadblock: deciphering conversational dynamics to accurately resolve "who said what and when." Decoupling and binding dynamic acoustic trajectories with visual identities and transcribed text is not only the core enabler for autonomous embodied agents, smart meeting transcription, and wearable AI assistants, but also the ultimate stress test for multimodal spatio-temporal synergy.
Current models suffer from an "illusion of competence" stemming from deeply flawed evaluation paradigms and an architectural perception gap. Existing benchmarks isolate speaker-related capabilities into siloed unimodal subtasks or inadvertently inject pervasive visual shortcuts—such as close-up portraits, visible handheld microphones, or single-person centered frames—allowing models to guess speaker identities from static visual cues without genuinely aligning acoustic signals with continuous lip motions. Architecturally, mainstream Omni models rely on sparse 1-2 fps visual sampling, which discards high-frequency inter-frame temporal residuals and destroys fine-grained visemes and micro-expressions. Naively increasing the frame rate of standard Vision Transformers triggers a catastrophic quadratic token explosion and drowns subtle lip movements in massive static scene redundancy.
Bridging this gap requires eliminating visual shortcuts while equipping Omni architectures with lightweight, high-frequency temporal modeling. Core idea: introduce the shortcut-free Visual-Registered Speaker Diarization and Recognition (VR-SDR) benchmark and develop HumanOmni-Speaker, a dual-stream architecture integrating a 25 fps Visual Delta Encoder that compresses inter-frame motion residuals into just 6 structured tokens per frame, achieving end-to-end lip reading and precise spatio-temporal identity binding without intrusive face cropping.
Method¶
Overall Architecture¶
HumanOmni-Speaker deconstructs multi-speaker video understanding into a dual-stream visual pathway balancing low-frequency spatial semantics and high-frequency motion dynamics. The input consists of raw uncropped panoramic video frames, continuous audio waveforms, and natural language identity registration queries (e.g., "Alice: A woman with long curly hair; Bob: A man in a black T-shirt"). The low-frequency branch adopts the Visual Base Encoder (1-2 fps) from Qwen2.5-Omni to extract static appearance and environmental context. The high-frequency branch deploys the novel Visual Delta Encoder (25 fps) to capture fine-grained mouth movements and spatial displacement trajectories.
The modality tokens—produced by the visual encoders, the continuous audio encoder, and the text tokenizer—are mapped into a unified latent space via dedicated projection layers. A shared LLM autoregressively outputs structured diarization transcripts containing identity labels, exact timestamps, and recognized content.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
A["Input: Raw Video Stream + Audio Waveform + Text Registration"] --> B["Dual-Stream Visual Decomposition"]
subgraph B["Dual-Stream Visual Decomposition"]
direction TB
B1["Visual Base Encoder<br/>1-2 fps for static semantics & identity"]
B2["Visual Delta Encoder<br/>25 fps for high-frequency motion residuals"]
end
B --> C["SVT Structured Spatio-Temporal Compression<br/>Hierarchical spatial & temporal conv into 6 tokens/frame"]
C --> D["Three-Stage Progressive Training & Modality Isolation<br/>Motion pre-training → isolated projector alignment → joint fine-tuning"]
D --> E["LLM Autoregressive Decoding<br/>Structured diarization output with timestamps & speaker IDs"]
Key Designs¶
1. Visual Delta Encoder: Fine-Grained Motion Primitives with Minimal Overhead To reconcile the conflict between high-frequency motion capture and catastrophic token explosion, the Visual Delta Encoder employs a three-stage lightweight pipeline. The Local Feature Perception stage utilizes a ResNet-18 backbone at 25 fps to extract local inter-frame motion residuals. The Structured Visual Tokenizer (SVT) then applies hierarchical \(7 \times 7\) spatial convolutions and a large-receptive-field temporal convolution (\(k = 63\)) to condense dense feature maps into exactly 6 structured tokens per frame. This preserves high-fidelity lip visemes and spatial trajectories while capping the token overhead at only +38% compared to a static baseline (vs. >10× explosion in naive 25 fps ViTs). Finally, a Transformer Encoder aggregates global context across frames into coherent temporal semantics. Crucially, shallow CNN layers naturally attend to the speaker's mouth without requiring explicit lip-cropping or face alignment preprocessing.
2. VR-SDR Benchmark: Rigorous Anti-Shortcut Evaluation Paradigm To dismantle the illusion of competence caused by static visual shortcuts, VR-SDR mandates end-to-end identity binding based solely on natural language descriptions without prior acoustic enrollment. Performance is evaluated under strictly identity-fixed metrics: - Identity-Fixed SA-WER (Speaker-Attributed Word Error Rate): Enforces absolute attribution to the vision-registered ID without allowing label permutation re-alignment: $\(\text{SA-WER} = \frac{\sum_{s \in \mathcal{S}} \text{WER}(\text{Ref}_s, \text{Hyp}_s)}{\sum_{s \in \mathcal{S}} |\text{Ref}_s|}\)$ - Identity-Fixed IER (Identification Error Rate): Strictly measures diarization precision against registered identities across missed speech \(\text{Miss}_s\), false alarms \(\text{FA}_s\), and speaker confusion \(\text{Conf}_s\) relative to ground-truth duration \(\text{Dur}_s\): $\(\text{IER} = \frac{\sum_{s \in \mathcal{S}} (\text{Miss}_s + \text{FA}_s + \text{Conf}_s)}{\sum_{s \in \mathcal{S}} \text{Dur}_s}\)$ The benchmark comprises 464 multi-person conversational videos (VR-SDR, 1.2 hours) and four diagnostic atomic tasks: Speech Recognition (SR), Speaker Verification (SV), Speaker Localization (SL), and Speaker Identification (SI). A meticulous manual filtering process establishes an explicit "Hard" split for SL and SI by purging all salient visual clues (such as close-ups, handheld mics, and single-person framing), compelling models to rely on true cross-modal temporal alignment.
3. Three-Stage Progressive Training and Modality Isolation Strategy To prevent dominant audio and text modalities from suppressing subtle visual motion residuals, the system is trained in three stages. In Stage 1 (Visual Delta Encoder Pre-training), the module is trained on large-scale datasets (AVSpeech, LRS2, LRS3, VoxCeleb2, AVA-ASD) using a multi-task objective combining face center-point regression and visual/audio-visual speech recognition (VSR/AVSR). In Stage 2 (Modality Alignment), both the Visual Delta Encoder and LLM are frozen; the Base Visual and Audio encoders are bypassed, feeding delta tokens exclusively through a lightweight MLP projector into the LLM to force translation of visemes into behavioral semantics without representational shortcuts. In Stage 3 (End-to-End Joint Fine-Tuning), 1.5 million unified instruction samples cover all atomic and diarization tasks; encoders remain frozen while projectors are trained, and the LLM undergoes hybrid tuning—full parameter fine-tuning on early layers for deep cross-modal fusion and LoRA adaptation on deeper layers to preserve general language fluency.
Loss & Training¶
In Stage 1, smooth \(L_1\) loss trains spatial tracking heads alongside CTC and cross-entropy loss for VSR/AVSR. In Stage 2, cross-modal alignment utilizes causal language modeling loss \(\mathcal{L}_{\text{LM}} = -\sum_{t} \log P(y_t | y_{<t}, X_{\text{delta}})\). In Stage 3, end-to-end multi-task instruction tuning optimizes generation with AdamW and cosine learning rate scheduling.
Key Experimental Results¶
Main Results¶
On the HumanOmni-Speaker Benchmark, HumanOmni-Speaker is evaluated against leading proprietary and open-source models (all values are error rates in %, lower is better):
| Model | Size | Speech Recognition SR (↓) | Speaker Verification SV (↓) | Speaker Localization SL (↓) | Speaker Identification SI-Hard (↓) | VR-SDR What (SA-WER) (↓) | VR-SDR When (IER) (↓) | Overall AVG (↓) |
|---|---|---|---|---|---|---|---|---|
| Gemini3-Pro | Closed | 1.39 | 5.2 | 12.8 | 30.5 | 36.6 | 36.3 | 36.5 |
| Qwen3-Omni-flash | Closed | 1.22 | 43.9 | 2.8 | 43.5 | 82.9 | 47.2 | 65.0 |
| OLA | 7B | 1.9 | 51.1 | 20.6 | 63.2 | 95.4 | 56.85 | 76.1 |
| VITA1.5 | 7B | 3.4 | 51.4 | 20.2 | 56.6 | 93.6 | 54.40 | 74.0 |
| Qwen2.5-Omni | 7B | 1.8 | 37.1 | 7.4 | 54.5 | 83.6 | 49.4 | 66.5 |
| Qwen2.5-Omni-SFT | 3B | 2.0 | 13.4 | 3.0 | 33.2 | 52.1 | 31.5 | 41.8 |
| HumanOmni-Speaker (Ours) | 3B | 1.9 | 13.2 | 0.8 | 21.1 | 47.1 | 28.5 | 37.8 |
Performance on Visual Speech Recognition (VSR) and Audio-Visual Speech Recognition (AVSR) across LRS2 and LRS3 (WER %): | Model Category | Model | Preprocessing | LRS2 VSR (↓) | LRS2 AVSR (↓) | LRS3 VSR (↓) | LRS3 AVSR (↓) | |---|---|---|---|---|---|---| | Specific VSR | AutoAVSR | Lip crop & align | 27.9 | 1.5 | 33.0 | 0.9 | | Specific VSR | AV-HuBERT Base | Lip crop & align | 31.2 | - | 34.8 | - | | LLM-AVSR | Llama-AVSR | Lip crop & align | - | - | 24.0 | 0.77 | | Omni Model | Qwen2.5-Omni | Raw video | - | 3.47 | - | 3.63 | | Omni Model (Ours) | HumanOmni-Speaker | Raw video (uncropped) | 29.8 | 1.36 | 33.4 | 0.76 |
Ablation Study¶
Ablation on multi-stream features and end-to-end design choices (Table 5 in paper):
| Configuration | Visual Base (1-2 fps) | Visual Delta (25 fps) | Audio Stream | Speaker Localization SL (%) (↓) | VR-SDR What (SA-WER %) (↓) | VR-SDR When (IER %) (↓) | Note |
|---|---|---|---|---|---|---|---|
| Pipeline-based (Whisper+Qwen3-VL) | - | - | - | - | 64.2 | 39.3 | Cascaded error accumulation |
| Visual Base only | Yes | No | Yes | 3.0 | 52.1 | 31.5 | Lacks high-frequency dynamics |
| Visual Delta only | No | Yes | Yes | 1.2 | - | - | Strong continuous spatial awareness |
| Full Model | Yes | Yes | Yes | 0.8 (-73%) | 47.1 (-9.6%) | 28.5 (-9.5%) | Optimal spatio-temporal binding |
Key Findings¶
- High-frequency residuals drive genuine cross-modal binding: Incorporating the Visual Delta Encoder into Qwen2.5-Omni drops Speaker Localization error from 3.0% to 0.8% (a 73.3% relative reduction) and slashes SI-Hard error from 33.2% to 21.1%. On the holistic VR-SDR task, SA-WER and IER improve from 52.1% and 31.5% to 47.1% and 28.5%, respectively.
- Raw full-frame input outperforms cascaded face cropping: Replacing raw video with bounding-box face crops from an off-the-shelf detector causes the SI-Hard error rate to surge from 21.1% to 38.0%, proving that cascaded detector failures and the loss of global body/head context severely impair disambiguation.
- Hyperparameter optimality: Operating at 25 fps is necessary to unlock fine-grained viseme modeling (2 fps fails completely, while 16 fps provides only basic recognition). Allocating 6 tokens per frame achieves the optimal saturation point for spatial tracking while avoiding unnecessary sequence bloat.
Highlights & Insights¶
- Native end-to-end lip reading in an Omni LLM: While prior visual speech recognition systems required fragile multi-stage face alignment and lip ROI cropping, HumanOmni-Speaker proves that spatio-temporal convolutional tokenization can natively learn visemes directly from uncropped raw video.
- Uncompromising anti-shortcut benchmark: By systematically exposing the "illusion of competence" in existing multi-modal models and establishing the Hard evaluation partition, VR-SDR establishes a trustworthy benchmark preventing static visual shortcuts from masquerading as cross-modal alignment.
- Decoupled dual-stream video tokenization: Separating spatial scene context (I-frame logic) from motion residuals (P/B-frame logic) provides a highly transferable architectural paradigm for other dynamic human-centric domains, including micro-expression recognition and sign language translation.
Limitations & Future Work¶
- Performance drop under multi-speaker overlap: When the number of simultaneous speakers increases from 2 to 3, all models (including Gemini3-Pro) suffer substantial degradation on SA-WER and IER, demonstrating that multi-party overlapping acoustic-visual separation remains an open frontier.
- Context window overhead in extended conversations: Even though 6 tokens per frame is highly compact, long multi-hour video streams at 25 fps eventually accumulate considerable sequence length, motivating future research into adaptive frame rates or state-space models (SSMs).
Related Work & Insights¶
- vs. Conventional Speaker Diarization (e.g., SpeakerLM, PyAnnote): Traditional systems rely on acoustic enrollment to extract voiceprint embeddings; VR-SDR performs identity binding purely from natural language visual descriptions.
- vs. Audio-Visual Speech Recognition (e.g., AutoAVSR, AV-HuBERT): Existing AVSR models function as isolated perception modules requiring tight lip cropping; HumanOmni-Speaker embeds robust lip-reading directly into a general-purpose conversational LLM.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneered shortcut-free VR-SDR benchmark and 6-token/frame 25 fps Visual Delta Encoder dual-stream architecture]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluation across 4 diagnostic atomic subtasks and holistic diarization with rigorous ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear problem formulation, rigorous motivation, and thorough analytical depth]
- Value: ⭐⭐⭐⭐⭐ [Crucial breakthrough for human-centric conversational AI, embodied agents, and intelligent meeting transcription]