Skip to content

EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/NYCU-NLP-Lab/EgoArgus
Area: LLM Agent
Keywords: First-person Assistant, Vision-Language Models, Modality Arbitration, Situational Intervention, Cross-modal Bias

TL;DR

EgoArgus introduces a dual-track benchmark comprising 6,978 real-video MCQA instances and 789 synthetic VISTA intervention episodes to evaluate vision-language models across context understanding, next-action prediction, and proactive intervention timing in five controlled video-dialogue relationship scenarios, uncovering that contemporary VLMs systematically collapse into text-dominant bias below random chance when verbal input contradicts visual reality.

Background & Motivation

With the rapid evolution of multimodal large language models (MLLMs) and vision-language models (VLMs), research on interactive intelligent systems is shifting from passive image and video perception toward proactive, interactive assistants. This paradigm shift is particularly consequential for egocentric and embodied settings, where smart glasses or wearable first-person robots accompany users throughout daily activities, continuously observing the environment from the user's perspective, comprehending ongoing procedural tasks, and following conversational dialogue. While foundational egocentric benchmarks such as Ego4D, QaEgo4D, and EgoPlan-Bench2 have catalyzed advancements in video question answering and physical planning, they largely assess visual reasoning in an isolated, noise-free vacuum without concurrent, multi-stream conversational inputs from the surrounding human context.

In real-world everyday assistance, however, an embodied VLM agent operates in an uncurated, heterogeneous sensory environment. The agent simultaneously receives two parallel information streams: a real-time first-person video feed depicting objective physical reality and a stream of natural speech dialogue spoken by the user and bystanders. Crucially, these two modalities are neither always synchronized nor uniformly dependable: user speech may supply complementary task context, linger on completely off-topic banter, or directly contradict physical reality due to user misperception, false memory, or simple oversights (such as asserting that the stove or water tap was turned off when the camera clearly shows it still running). Under such cross-modal friction, a reliable situational assistant cannot indiscriminately treat all inputs as equally trustworthy; it must arbitrate which modality bears the definitive evidence, suppress misleading verbal claims, and exercise disciplined restraint by remaining silent during benign tasks while proactively intervening at the exact critical moment when safety or operational failure threatens.

Prior egocentric benchmarks fail to explicitly isolate the semantic interplay between video and user dialogue, nor do they evaluate calibrated intervention timing and negative no-assistance control scenarios simultaneously. To address this fundamental gap between static multimodal perception and dynamic, reliable first-person assistance, this work introduces EgoArgus. Core idea: systematically dissect egocentric assistance capabilities into context understanding, next-action prediction, and proactive intervention decisions across five controlled dialogue-video relationship scenarios, establishing a rigorous diagnostic benchmark to expose and analyze cross-modal arbitration and timing calibration in frontier vision-language models.

Method

Overall Architecture

The EgoArgus benchmark comprises a dual-track architecture unifying an Understanding Part and a Decision Part to assess the entire spectrum of egocentric assistance. The end-to-end framework spans four interconnected stages: controlled scenario taxonomy and corpus construction, dual-track benchmark synthesis with human-in-the-loop verification, multi-layer linear probing diagnostics, and a five-dimensional failure taxonomy audit.

At test time, candidate models are fed the first-person continuous video stream, the synchronized multi-turn conversational transcript, and dialogue timestamps. In the understanding track, models perform four-way multiple-choice question answering (MCQA) to evaluate context perception and next-action prediction under controlled conversational distractors. In the decision track, models receive unprompted raw streams and must autonomously decide whether assistance is required, predict the precise intervention timestamp, summarize the issue, and produce executable, grounded assist steps.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Input Streams<br/>First-Person Video + Synchronized Multi-Turn Dialogue + Timestamps"] --> B["Controlled Scenario Modeling & Corpus Synthesis"]
    B --> C["Dual-Track Benchmark & Human-in-the-Loop Review"]
    C --> D["Context Understanding & Next-Action Prediction<br/>MCQA: 6,978 Real-Video Instances"]
    C --> E["VISTA Proactive Intervention & Timing Benchmark<br/>Assist-Step: 789 Synthetic First-Person Episodes"]
    D --> F["Layer-Wise Linear Probing & Representation Separation"]
    E --> G["Five-Fold Intervention Failure Taxonomy & Alignment Audit"]
    F --> H["Comprehensive Situational Assistant Reliability Report"]
    G --> H

Key Designs

1. Controlled Scenario Modeling & Corpus Synthesis: Resolving the confounding of cross-modal semantic relationships by formalizing a five-scenario interaction taxonomy

To overcome the inability of existing benchmarks to isolate whether model failures stem from visual blindness or conversational misguidance, EgoArgus formalizes five distinct video-dialogue relationship scenarios: (1) Multimodal Grounded, where dialogue provides instructions or relational attributes while video grounds referenced physical objects and actions, requiring cooperative synthesis; (2) Contradictory, where visual evidence determines ground truth while dialogue provides an explicit factual contradiction; (3) Video-Grounded On-Topic, where dialogue is topically related to the task but provides no answer-bearing information; (4) Video-Grounded Off-Topic, where dialogue consists of entirely unrelated chitchat; and (5) Text-Grounded, where the query and answer depend strictly on conversational intent while the video serves as an irrelevant visual distractor.

To instantiate these scenarios for the understanding track, the authors preserve original video question pairs from EgoPlan-Bench2, QaEgo4D, and conversational intents from MIntRec. Gemini-3.1-Pro is prompted to synthesize candidate dialogues under strict constraints (e.g., generating misleading claims aligned with an incorrect distractor choice, or generating topically relevant text that strictly omits the gold answer), ensuring that the underlying visual QA task remains constant while only the cross-modal semantic relation varies.

2. Dual-Track Benchmark & Human-in-the-Loop Review: Tackling the sparsity of high-risk physical interventions and the absence of no-assistance controls via complementary real and synthetic suites

Because real-world egocentric daily video captures very few dense hazard interventions due to safety, privacy, and collection constraints, EgoArgus deploys a complementary dual-track paradigm: The Understanding Track compiles 6,978 real-video MCQA examples (3,000 from EgoPlan-Bench2, 3,000 from QaEgo4D, and 978 from MIntRec) to thoroughly test context understanding and next-action prediction. To guarantee semantic fidelity and eliminate LLM generation artifacts, 16 trained human annotators conduct three-way independent reviews per candidate, performing majority voting to validate that scenario constraints are satisfied and correcting ambiguous gold answers. The Decision Track leverages VISTA (Liu et al., 2026a), a controllable first-person physical simulation platform, to generate 789 first-person assistant episodes covering identical relationship scenarios alongside 239 benign No-Assistance controls. Over 2,917 human audit events (encompassing 565 attempt-review events and 447 script revisions) prune visual artifacts, viewpoint breaks, and seed mismatches. Each verified episode provides oracle metadata specifying assistance necessity, the earliest viable intervention timestamp \(t_{\text{oracle}}\), the core issue, and actionable remediation steps.

3. Layer-Wise Linear Probing & Representation Separation: Diagnosing the internal phase transition of answer preferences across decoder depth

To determine whether cross-modal failures arise from fused, indistinguishable hidden states or from late-stage decision-level text subservience, the authors train independent linear probes across each decoder layer of four open-weight models (Qwen3.5-2B, InternVL3.5-4B, Cosmos-Reason2-8B, and Molmo2-8B) on 1,000 balanced instances (200 per scenario). For layer \(l\), the probe receives the normalized last-token hidden state \(h_l \in \mathbb{R}^d\) and predicts the model's own soft output distribution over options (A, B, C, D) using soft cross-entropy: $\(\mathcal{L}_{\text{probe}} = -\sum_{i \in \{A, B, C, D\}} P_{\text{VLM}}(y=i \mid X) \log \left(\text{Softmax}(W_l h_l + b_l)_i\right)\)$ Across all evaluated architectures, probe accuracy remains near baseline through early-to-middle layers before executing a steep phase transition at ~67%–69% relative decoder depth (e.g., jumping from 49.5% at Layer 15 to 94.5% at Layer 16 in Qwen3.5-2B, and from ~65% to >95% at Layer 25 in 36-layer models). SVD projections of visual versus dialogue token representations reveal that while the two modalities become clearly separated by intermediate layers, the model's ultimate decision head predominantly heeds the text stream, causing catastrophic failure under contradictory dialogue.

4. Five-Fold Intervention Failure Taxonomy & Alignment Audit: Unmasking pathological assistant policies beyond aggregate F1 scores

A single aggregate decision metric frequently conceals severe behavioral pathologies in daily assistance. The benchmark categorizes intervention failures into five distinct modes: (1) Grounding Errors, failing to identify physical scene or dialogue evidence; (2) Step Errors, proposing vague, infeasible, or ungrounded assist actions; (3) Timing Errors, intervening either prematurely before sufficient evidence accumulates or tardily after the event concludes; (4) Missed Interventions (False Negatives), remaining silent when hazardous or non-safety assistance is required; and (5) No-Assist False Alarms (False Positives), aggressively interrupting benign everyday actions. Timing precision is quantitatively tracked via Mean Absolute Error (MAE) and signed latency \(\Delta t = t_{\text{pred}} - t_{\text{oracle}}\), providing an empirical framework to audit why uniform visual attention reweighting and noise-aware preference optimization (NaPO) fail to resolve cross-modal bias.

Benchmark & Evaluation Protocol

The benchmark evaluates seven leading vision-language models: Molmo2-8B, Qwen3.5-2B, InternVL3.5-4B, Gemini-3.1-Flash-Lite, Cosmos-Reason2-8B, Qwen3.5-Plus (02-15), and MiMo-V2-Omni (with Gemini-3.1-Pro serving as an additional strong reference). Evaluation metrics are structured into two distinct domains: - Understanding Part: Accuracy (%) on four-choice MCQA across tasks (EgoPlan-Bench2, MIntRec, QaEgo4D) and across the five semantic scenarios. - Decision Part: Overall Assistance-Decision F1, Timing MAE (seconds), Signed Timing Error \(\Delta t\) (seconds), scenario-specific F1 scores across six conditions (Multimodal, Contradictory, Video On-Topic, Video Off-Topic, Text-Grounded, No-Assistance), and three critical failure rates: Safety False Negatives (Safety FN), Non-Safety False Negatives (Non-safety FN), and No-Assistance False Positives (No-assist FP).

Key Experimental Results

Main Results

The performance of frontier vision-language models on the VISTA decision-part benchmark, along with timing precision and failure distribution, is summarized in Table 1 (transcribed exactly from Table 2 of the paper):

Model Input Mode Overall F1 ↑ MAE ↓ (s) \(\Delta t\) (s) Multi. F1 ↑ Contra. F1 ↑ Vid On F1 ↑ Vid Off F1 ↑ Text F1 ↑ No Asst. F1 ↑ Safety FN ↓ Non-saf. FN ↓ No-asst. FP ↓
Gemini 3.1 Flash Lite Full-video 0.8117 2.7420 -2.4970 0.9395 0.9042 0.8049 0.7431 0.9463 0.3885 0.1445 0.2443 0.3173
MiMo V2 Omni Full-video 0.7500 2.6340 -2.3420 0.9194 0.8681 0.8171 0.7046 0.7031 0.3051 0.1992 0.3969 0.2620
Qwen3.5 Plus 02-15 Full-video 0.7232 2.5050 -1.9280 0.9340 0.9101 0.7249 0.6087 0.4444 0.4418 0.3008 0.4695 0.1144
InternVL3.5 4B Sampled 0.5979 3.7030 -0.4040 0.9091 0.8587 0.8571 0.1935 0.0909 0.2483 0.4141 0.5687 0.3506
Cosmos Reason2 8B Sampled 0.5572 2.7840 -1.7310 0.7283 0.7577 0.6131 0.5165 0.1897 0.3095 0.4023 0.7595 0.1070
Molmo2 8B Sampled 0.5375 4.0190 -3.9950 0.8265 0.8603 0.7655 0.0854 0.1251 0.1961 0.4922 0.6107 0.3985
Qwen3.5 2B Full-video 0.1920 3.7840 -3.7840 0.3459 0.1509 0.2095 0.0854 0.2203 0.0000 0.8438 0.9237 0.0037

Ablation & Diagnostic Studies

To empirically confirm that cross-modal distraction drives catastrophic accuracy degradation, Table 2A reports the oracle distractor-removal experiment on 1,000 balanced instances (200 per scenario), while Table 2B examines test-time uniform visual attention reweighting and training-time preference alignment (NaPO) on Qwen3.5-2B:

Table 2A: Full-Input vs. Oracle Distractor-Removal Accuracy on Real-Video MCQA (%)

Model Input Regime Multimodal Contradictory Off-topic On-topic Text-grounded Mean Acc.
Qwen3.5-2B Full Input 64.0 16.0 48.5 50.5 73.5 50.5
Drop Distractor 64.0 57.0 (+41.0) 49.5 (+1.0) 49.5 (-1.0) 73.0 (-0.5) 58.6 (+8.1)
InternVL3.5-4B Full Input 73.5 13.0 50.0 49.5 76.0 52.4
Drop Distractor 73.5 56.0 (+43.0) 49.0 (-1.0) 49.0 (-0.5) 77.0 (+1.0) 60.9 (+8.5)
Cosmos-Reason2-8B Full Input 77.0 10.5 49.0 54.5 81.0 54.4
Drop Distractor 77.0 51.5 (+41.0) 50.0 (+1.0) 50.0 (-4.5) 82.5 (+1.5) 62.2 (+7.8)
Molmo2-8B Full Input 81.5 27.5 54.5 56.0 79.5 59.8
Drop Distractor 81.5 61.5 (+34.0) 56.5 (+2.0) 56.5 (+0.5) 81.0 (+1.5) 67.4 (+7.6)

Table 2B: Mitigation Baselines on Qwen3.5-2B Real-Video MCQA (%)

Scenario Base Attn \(\epsilon=-10\) Attn \(\epsilon=-5\) Attn \(\epsilon=5\) Attn \(\epsilon=10\) NaPO Pipeline Shift \(\Delta\) (NaPO vs Base)
Multimodal 69.4 67.5 67.3 68.2 36.8 67.6 -1.8
Contradictory 18.4 15.9 15.9 16.5 18.5 15.7 -2.7
Off-topic 48.4 44.4 44.1 44.2 26.6 44.4 -4.0
On-topic 50.7 47.7 47.5 46.8 27.6 47.5 -3.2
Text-grounded 73.7 74.1 74.4 72.6 38.7 75.4 +1.6
Overall Mean 51.3 - - - - 48.9 -2.5

Key Findings

  • Contradictory dialogue drives accuracy below random chance: In the understanding track, mean model accuracy collapses from 63.0% on multimodal-grounded items down to 18.5% under contradictory dialogue. Every single model except Molmo2-8B (24.9%) falls well below the 25% random-guess threshold, proving that misleading dialogue does not simply inject noise but actively inverts model judgment.
  • Asymmetric text dominance: Models attain a high mean accuracy of 71.0% on text-grounded examples despite noisy video backgrounds, demonstrating an innate propensity to rely on linguistic priors while failing to ground decisions in visual evidence when text conflicts.
  • Oracle distractor isolation is highly scenario-specific: Eliminating contradictory speech surges MCQA accuracy by 34.0–43.0 points across all architectures; conversely, eliminating on-topic dialogue induces performance drops of up to -4.5 points, demonstrating that answerless conversational context provides vital scene priors that cannot be blindly stripped.
  • Abrupt late-stage representation phase transition: Layer-wise linear probes uncover that option preference decodability is negligible across the first 60% of decoder layers, suddenly leaping from ~50% to >94% at 67%–69% depth. Modalities are separated in latent space, yet the final decision layer systematically capitulates to linguistic tokens.
  • Intervention policy exhibits severe precision-recall and timing dilemmas: Strong models (e.g., Gemini-3.1-Pro) achieve high safety recall but suffer from an unbearable no-assist false alarm rate of 56.8%. Furthermore, all models exhibit negative signed timing errors (\(\Delta t\) between -0.4s and -4.0s), indicating a pervasive premature-intervention bias before sufficient visual context unfolds.

Highlights & Insights

  • First systematic framework formalizing cross-modal semantic relationships in egocentric assistance: By deconstructing multi-stream interactions into five distinct modalities alongside negative no-assistance episodes, EgoArgus dismantles the naive assumption that multimodal fusion is universally synergistic.
  • Empirical refutation of heuristic attention reweighting and standard preference alignment: Neither brute-force visual attention boosting \(\epsilon\) nor NaPO preference optimization resolves the cross-modal arbitration dilemma, with NaPO actually depressing overall accuracy by 2.5%, highlighting the fundamental need for context-conditioned, adaptive routing.
  • Critical guidance for edge wearable deployment: The demonstration that 2B–4B models suffer severe step-planning errors and premature triggers underscores that edge wearable devices require dedicated, lightweight visual-fidelity verification gates prior to late-layer action decoding.

Limitations & Future Work

  • Closed-world MCQA constraint in understanding evaluation: While four-choice question answering enables mathematically rigorous random-chance baselines and precise distractor calibration, it does not fully encompass open-ended conversational generation encountered in real-world dialogue.
  • Synthetic visual simulation gap in VISTA: Although VISTA allows dense sampling of hazardous scenarios and fine-grained negative controls, generated synthetic videos exhibit slight distribution shifts relative to real wearable cameras in sensor noise, hand-object occlusions, and complex natural lighting.
  • Future architectural avenues: Future efforts should focus on integrating dynamic cross-modal confidence estimators, multimodal gating adapters, and gaze-tracking coordination to achieve sub-second, calibrated intervention timing.
  • vs Ego4D / EgoPlan-Bench2 / QaEgo4D: Conventional egocentric datasets evaluate visual planning and grounded QA assuming pristine, isolated video input. EgoArgus treats conversational user dialogue as an active, potentially contradictory co-modality, introducing cross-modal arbitration into first-person benchmarking.
  • vs HoloAssist / Ego-EXTRA: While prior work explores instructor-trainee interaction, they collect naturalistic collaborative dialogues without adversarial semantic perturbations or negative control conditions. EgoArgus isolates specific failure modes via five controlled scenarios and unprompted intervention decisions.
  • vs ProactiveVideoQA / OVO-Bench: Existing proactive video benchmarks evaluate stream-based QA but largely overlook multi-turn conversational arbitration and explicit negative silence controls. EgoArgus is the only benchmark spanning all six axes: first-person video, dialogue, intervention decision, timing, assist steps, and no-assistance handling.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes the first controlled egocentric situational assistance benchmark formalizing five cross-modal relationship scenarios and proactive intervention timing.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluates 7 leading VLMs across 6,978 real-video MCQA items and 789 synthetic VISTA episodes, complemented by layer-wise probing and debiasing ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured and lucid, supported by rich empirical figures and deep mechanistic insights into late-layer representation collapse.
  • Value: ⭐⭐⭐⭐⭐ Delivers vital empirical benchmarks and design principles for building dependable, next-generation smart glasses and first-person AI assistants.