Skip to content

EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://egocentricvoice.github.io/
Area: LLM Reasoning
Keywords: proactive assistant, egocentric vision, spoken dialogue, full-duplex MLLM, direct preference optimization

TL;DR

Addressing the critical absence of proactive spoken guidance benchmarks and end-to-end training paradigms from streaming first-person inputs, EgoVoice reconstructs HoloAssist via sound source separation and duration-controlled speech resynthesis, presenting an end-to-end full-duplex multimodal framework optimized via SFT and length-normalized DPO that intervenes with timely speech instructions that conclude before user actions begin.

Background & Motivation

With the rapid emergence of smart glasses and augmented reality (AR) wearables, intelligent personal assistants are shifting from conventional reactive turn-taking interfaces toward continuous, proactive first-person collaboration. In physical procedural endeavors such as assembling modular furniture, troubleshooting complex machinery, or cooking, users are frequently occupied with their hands and eyes, experiencing substantial cognitive loads. Crucially, a user may not anticipate an imminent procedural mistake or may be unable to pause an ongoing manual process to explicitly query an assistant. If a conversational system requires explicit verbal prompting before offering help, it invariably misses the critical temporal window for preempting errors. This operational necessity calls for always-on proactive egocentric spoken assistance, wherein an agent observes continuous first-person video and audio streams, independently deciding whether to remain silent, when to speak, and what verbal instruction to offer.

Despite recent progress in omni-modal foundational models, current systems suffer from deep-seated architectural and empirical blind spots. First, multimodal perception, full-duplex conversational reasoning, and proactive streaming intervention have historically been researched in isolation: commercial voice agents rely strictly on voice activity detection (VAD), remaining silent during unprompted manual work; meanwhile, prior proactive video agents generate exclusively textual outputs. Naively applying text-to-speech (TTS) pipelines over text-based video dialog corpora incurs severe physical timeline mismatches—unlike text tokens which are ingested instantaneously by an interface, spoken speech exhibits non-zero physical duration. If an utterance is synthesized at the text timestamp, the user will already be deep into or finished with their physical action before the synthesized speech concludes; shifting speech earlier introduces artificial collisions with preceding dialogue and precedes visual context. Second, real-world physical environments are saturated with acoustic noise, tools, and background sounds, causing off-the-shelf full-duplex models like MiniCPM-o 4.5 to default to third-person scene narration rather than concise, second-person instructional guidance.

To overcome these structural hurdles, this paper takes inspiration from real-world human instructor-operator interactions, leveraging natural human communication timing while addressing severe acoustic artifacts. The raw multi-speaker recordings in HoloAssist capture authentic human instructional pacing and ambient noise, yet their mono channel mixture and pitch-shifted anonymization render them unusable for natural acoustic modeling. Core idea: by combining prompt-guided source separation and duration-controlled zero-shot speech resynthesis, this work converts HoloAssist into the first continuous proactive egocentric voice benchmark, and subsequently aligns a streaming full-duplex multimodal LLM via chunk-wise SFT and length-normalized DPO to master both unprompted intervention timing and speech duration constraints.

Method

Overall Architecture

The EgoVoice framework is organized around four unified pillars: high-fidelity streaming data reconstruction from raw recordings, an action-coupled full-duplex evaluation protocol, supervised fine-tuning with acoustic-semantic gradient detachment, and length-normalized direct preference optimization (DPO) for timing alignment. The model continuously ingests egocentric video frames sampled at 1 fps alongside one-second streaming audio chunks (composed of ambient acoustic background and user vocalizations), autoregressively outputting either a silent token (LISTEN) or a proactive intervention token (SPEAK) paired with discrete speech representations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Egocentric Multimodal Stream<br/>1 fps video + ambient audio + user speech"] --> B["Stage 1: Source Separation & Duration-Controlled Resynthesis<br/>SAM-Audio isolation + IndexTTS2 constrained synthesis"]
    B --> C["Stage 2: Action-Coupled Full-Duplex Evaluation Protocol<br/>Asymmetric Hungarian matching + end-vs-action timing"]
    C --> D["Stage 3: Full-Duplex SFT Behavioral Cloning<br/>MiniCPM-o 4.5 chunked tuning + gradient detachment"]
    D --> E["Stage 4: Length-Normalized DPO Timing Alignment<br/>High vs. low F1 preference pairs + temporal reward"]
    E --> F["Output: Proactive Spoken Guidance<br/>Speech completes prior to next physical action"]

Key Designs

1. Source Separation & Duration-Controlled Resynthesis: Resolving Acoustic Mismatch and Duration Drift

Directly converting existing text-based video instruction corpora into spoken speech introduces catastrophic temporal misalignment: text tokens assume instantaneous visibility, whereas speech takes seconds to play back. Placing synthetic speech at the original text timestamp causes the user to execute the action while the speech is still playing; shifting it earlier risks colliding with prior dialogue or issuing advice before visual context appears. To ground assistance in genuine human interaction timing, the authors utilize HoloAssist, which contains real human instructor speech, but resolve its mono-channel mixing and pitch-shifted anonymization through a rigorous pipeline. They prompt SAM-Audio with speech timestamps to isolate human voices, retaining the acoustic residual as the background audio channel, and use Silero VAD to discard any session with speech leakage. They then normalize the transcripts using gpt-5.4-mini (correcting syntax and removing 9.0% recording-related opening/closing remarks) and resynthesize speech with IndexTTS2 using disjoint LibriSpeech reference speakers. Crucially, each utterance is synthesized up to 20 times, discarding candidates longer than the original instructor's utterance duration, and selecting the candidate with zero WER via Qwen3-ASR-1.7B. The resulting corpus preserves authentic human intervention timing and background acoustics across 1,249 training sessions (91.7 hours) and 170 held-out evaluation sessions (12.9 hours).

2. Action-Coupled Full-Duplex Evaluation Protocol: Decomposing Onset and End-of-Utterance Timings

Evaluating proactive speech purely by start-time timestamps fails in physical environments, where assistance must conclude before the user commits to a motor action. The authors design a comprehensive protocol evaluated directly on the delivered acoustic channel (text-asr). First, onset timing is measured via Hungarian matching using an asymmetric tolerance window of \(-2.5\text{s}\) (early) to \(+1.5\text{s}\) (late), capturing that early advice is far less damaging than delayed advice. Second, they introduce the end-vs-action metric on an action-linked subset, calculating the temporal gap between the onset of the user's next physical action and the assistant's speech termination: $\(\Delta t_{\text{end-vs-act}} = t_{\text{action\_start}} - t_{\text{speech\_end}} = (t_{\text{action\_start}} - t_{\text{speech\_start}}) - \text{duration}\)$ A positive value signifies that guidance completed before action initiation; a negative value indicates that the assistant was still speaking during execution. To isolate procedural directives from non-directive conversational tokens, gpt-5.5 classifies reference utterances through 5-way self-consistency, linking directives to subsequent actions within \(T=5\text{s}\) (yielding 942 linked utterances across 170 sessions). Word-level alignments from Qwen3-ForcedAligner-0.6B on the transcribed audio provide exact physical utterance boundaries.

3. Full-Duplex SFT Behavioral Cloning: Chunked Decision Modeling and Acoustic-Semantic Gradient Detachment

Off-the-shelf full-duplex backbones default to passive chat and struggle with timing discipline. The authors adapt MiniCPM-o 4.5 by processing 60-second sliding windows sliced into one-second granularity chunks \(s_t = (v_t, a_t, z_t, w_t, c_t)\), where \(v_t\) is the video frame, \(a_t\) is the composite input audio (ambient noise + user speech), \(z_t \in \{\text{LISTEN}, \text{SPEAK}\}\) is the decision token, \(w_t\) is the normalized text, and \(c_t\) denotes discrete S3 speech tokens. LoRA adapters (\(r=32, \alpha=64\)) are applied to both the LLM backbone and the 0.3B speech-token decoder while freezing audio-visual encoders. The joint objective is defined as: $\(\mathcal{L}_{\text{SFT}} = \mathcal{L}_{\text{LLM}} + \lambda \cdot \mathcal{L}_{\text{S3}}\)$ with \(\lambda = 0.1\), where \(\mathcal{L}_{\text{LLM}}\) governs autoregressive cross-entropy over decision tokens, BPE text tokens, and end-of-turn markers, and \(\mathcal{L}_{\text{S3}}\) optimizes cross-entropy over S3 speech tokens. Crucially, gradients from \(\mathcal{L}_{\text{S3}}\) are detached at the LLM hidden states, preventing low-level acoustic loss signals from corrupting high-level decision semantics and language generation.

4. Length-Normalized DPO Timing Alignment: Contrastive Self-Calibration for Concise Interventions

Supervised fine-tuning provides strong foundational behavior, but the open-ended nature of streaming assistance allows suboptimal verbosity and premature triggering. To refine temporal precision, the authors apply direct preference optimization over candidate outputs generated by the frozen SFT checkpoint \(\pi_{\text{ref}}\). For each training window, \(N=10\) candidate trajectories are sampled and scored using onset F1. The candidate with the highest F1 serves as the chosen response \(y^+\), while a rejected candidate \(y^-\) is sampled from the bottom three ranks, enforcing a minimum margin of \(\Delta \text{F1} \ge 0.10\) across 5,335 preference pairs. Training optimizes length-normalized DPO: $\(\mathcal{L}_{\text{DPO}} = -\mathbb{E} \left[ \log \sigma \left( \frac{\beta}{|y^+|} \rho_{y^+} - \frac{\beta}{|y^-|} \rho_{y^-} \right) \right]\)$ where \(\rho_y = \log \pi_\theta(y|x) - \log \pi_{\text{ref}}(y|x)\), and \(\beta = 0.1\). LoRA is applied exclusively to the LLM backbone, holding the speech decoder frozen. This ensures that preference optimization strictly penalizes mistimed turns and verbose rambling without destabilizing acoustic feature synthesis.

Loss & Training

Supervised fine-tuning utilizes a cosine learning rate schedule (peak lr = 2e-4 with 5% warmup), effective batch size of 32, and bf16 mixed precision with FlashAttention-2 over 5 epochs (2,140 steps) on 13,676 windows. DPO fine-tunes for 1 epoch (334 steps) with a learning rate of 5e-6 and batch size of 16. Computation utilizes NVIDIA A6000, RTX PRO 6000, and B200 GPUs.

Key Experimental Results

Main Results

Primary evaluation results across 170 held-out benchmark sessions on the delivered audio channel (text-asr):

Model Model Type Onset Prec. Onset Rec. Onset F1 Cosine Sim. LLM Judge: Correctness LLM Judge: Promptness LLM Judge: Efficiency LLM Judge: Overall End-vs-action (s) Mean / Median
Gemini Live (2.5-flash) Commercial API 0.379 0.019 0.022 0.448 2.09 2.39 2.67 1.94 -3.20 / -2.26
gpt-realtime Commercial API 0.285 0.024 0.030 0.500 2.14 2.60 2.52 2.10 -7.40 / -7.46
gpt-realtime-2 Commercial API 0.256 0.019 0.024 0.456 2.28 2.40 2.10 2.14 -16.1 / -16.8
ProAssist (8B, Text-only) Open Baseline 0.480 0.158 0.195 0.594 2.34 2.63 2.99 2.06 —
ProAssist + ASR/TTS Cascade Cascaded 0.479 0.158 0.195 0.564 2.13 2.50 2.87 1.92 -0.44 / -1.56
MiniCPM-o 4.5 (Zero-shot) Open Backbone 0.259 0.108 0.113 0.519 1.54 1.63 1.73 1.41 -4.69 / -0.40
EgoVoice-SFT Ours 0.467 0.235 0.274 0.684 2.41 2.64 2.64 2.23 +1.57 / +1.52
EgoVoice-DPO Ours 0.482 0.285 0.324 0.649 2.35 2.55 2.42 2.18 +1.75 / +1.62
Human Instructor Ground Truth — — — — — — — — +1.69 / +1.38

Ablation Study

Ablation of key training choices and DPO loss variants across the 170 benchmark sessions (text-asr):

Configuration / Variant Stage Targeted Onset Prec. Onset Rec. Onset F1 Cosine Sim. LLM Judge (Overall) End-vs-action (s) Mean / Median
EgoVoice-SFT (Full) SFT Reference 0.467 0.235 0.274 0.684 2.23 +1.57 / +1.52
w/o Recipe in System Prompt SFT Ablation 0.398 0.223 0.240 0.621 1.81 +2.32 / +2.09
w/o Gradient Detachment SFT Ablation 0.435 0.246 0.276 0.680 2.18 +2.19 / +1.86
EgoVoice-DPO (Length-norm., \(\beta=0.1\)) DPO Selected 0.482 0.285 0.324 0.649 2.18 +1.75 / +1.62
Vanilla DPO (\(\beta=0.1\)) DPO Variant 0.458 0.261 0.298 0.651 2.25 +1.14 / +1.40
SimPO (Reference-free) DPO Variant 0.457 0.309 0.328 0.641 2.18 +1.86 / +1.77
Length-norm. DPO (\(\beta=0.5\)) DPO Penalty 0.401 0.371 0.345 0.617 2.04 +1.62 / +1.62
F1 + Cosine Joint Reward DPO Reward 0.451 0.205 0.248 0.674 2.16 +1.51 / +1.53

Key Findings

  • Closed-source voice APIs fail in unprompted proactive settings: Gemini Live and OpenAI Realtime models achieve an onset F1 of at most 0.03 because their server-side VAD architecture requires active user vocalization. Furthermore, when they do trigger, their excessively long speech turns (gpt-realtime-2 averages 51.6 words and 20.39 seconds duration) cause severe end-vs-action delays of -7.40s to -16.1s.
  • Untuned multimodal models suffer from severe narration bias: Zero-shot MiniCPM-o 4.5 exhibits only 30.1% instructional guidance, spending 21.0% on third-person scene narration ("The person is unscrewing...") and 49.0% on ungrounded chit-chat, resulting in a low LLM Overall score of 1.41. In contrast, EgoVoice shifts instructional guidance to 64.2%–65.7% and cuts scene narration down to 5.5%, directly matching the human instructor benchmark (5.6%).
  • SFT and DPO exhibit clear complementary strengths: SFT excels at content fidelity and semantic precision (highest cosine similarity of 0.684 and judge correctness of 2.41), while DPO sharply optimizes intervention recall and onset timing (\(F1\) jumps from 0.274 to 0.324, \(p = 0.0007\)).
  • End-vs-action parity with human instructors: EgoVoice-SFT (+1.57s / +1.52s) and EgoVoice-DPO (+1.75s / +1.62s) are the only automated models achieving positive end-vs-action margins, closely replicating human instructor timing (+1.69s / +1.38s). In contrast, the cascaded ProAssist system overshoots user actions in 62.0% of interventions due to unconstrained utterance duration.
  • Strict procedural ordering preserved: Evaluating matched guidance against reference instructions via Kendall's rank correlation (\(\tau_b\)) reveals that EgoVoice-DPO achieves 0.868 and SFT achieves 0.841, outperforming MiniCPM-o 4.5 (0.749) and ensuring that guidance strictly adheres to chronological assembly logic.

Highlights & Insights

  • Uncovering the "Speech Duration Dilemma" in Proactive Systems: Prior proactive vision systems evaluated only timestamp accuracy (onset). EgoVoice demonstrates that in physical tasks, speech duration dictates physical feasibility: deciding when to speak is trivial compared to finishing speech before the operator initiates the action.
  • Acoustically Pristine Data Extraction from Artifact-Ridden Video: The combination of SAM-Audio prompt-based background separation and IndexTTS2 duration-constrained resynthesis provides a blueprint for cleaning real-world egocentric interactions without losing natural ambient audio.
  • Single-Signal Preference Optimization Outperforms Composite Rewards: Attempting to balance timing (F1) and content (cosine similarity) concurrently during DPO introduces reward contention that degrades both axes (F1 drops to 0.248). Optimizing strictly over temporal alignment yields superior downstream task performance.

Limitations & Future Work

  • Procedural Domain Restriction: The training corpus derives exclusively from HoloAssist assembly tasks. Generalization to unstructured culinary workflows, high-speed sports, outdoor navigation, or socially sensitive interactions remains unverified.
  • One-Second Chunking Bottleneck: The temporal discretization inherited from MiniCPM-o 4.5 limits intervention precision to 1.0-second boundaries, precluding sub-second reflexes needed in critical safety warnings.
  • Lack of Bidirectional Barge-in Modeling: Current evaluation assumes uninterrupted playback. Developing full-duplex dynamic turn-taking that accommodates user interruptions or allows the assistant to yield mid-sentence is vital for deployment.
  • Backbone Repetition and Language Drift: A small percentage of utterances suffer from generative instability inherited from MiniCPM-o 4.5 (47 of 1,090 utterances exhibited repetition loops, and 13 experienced language switching), warranting broader foundation-level pretraining.
  • vs ProAssist (EMNLP 2025): While ProAssist formulated proactive video dialogue, it operates strictly in the text modality. When augmented with an off-the-shelf ASR/TTS cascade, its verbose turn structure (16.7 words/turn) causes 62.0% of spoken interventions to finish after the user begins the action. EgoVoice's end-to-end multimodal modeling maintains concise turns (7.8 words/turn), cutting late interventions to 28.2% and lifting onset F1 from 0.195 to 0.324.
  • vs MiniCPM-o 4.5 (2026): MiniCPM-o 4.5 provides full-duplex omni-modal foundations but lacks task orientation, squandering bandwidth on observational narrations (Overall score 1.41). EgoVoice adapts its dual-token interface to goal-driven procedural assistance, lifting the Overall score to 2.23 and aligning speech completion with human instructor standards.
  • Insights: Assistive AI deployed on wearable devices must account for physical speech delivery latency. Training objectives for voice agents should integrate speech length constraints directly into reinforcement learning rewards.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering work establishing proactive spoken assistance from continuous multimodal first-person streams, introducing duration-constrained audio synthesis and end-vs-action evaluation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across proprietary speech APIs, cascaded systems, and open backbones, featuring extensive temporal decomposition, LLM-as-a-judge, MTurk human evaluation, and rigorous ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing that cleanly articulates the physical and temporal tensions of spoken dialogue in physical procedural tasks.
  • Value: ⭐⭐⭐⭐⭐ High-impact contribution providing the first open dataset, benchmark, and trained full-duplex models for proactive AR companions.