GuideMe: Benchmarking Multi-Domain Task Guidance and Intervention in Streaming Video¶

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://guideme-benchmark.github.io
Area: Video Understanding
Keywords: Multimodal Large Language Models, Interactive Task Guidance, Streaming Video, Error Detection & Correction, Benchmark

TL;DR¶

Constructs GuideMe, the first multi-domain streaming video benchmark for closed-loop interactive task guidance (comprising 2,458 long videos and 47,775 interaction samples), and introduces a tripartite evaluation framework combining temporal-semantic bipartite matching, intervention timing classification, and LLM-as-a-Judge to expose a critical capability asymmetry: current MLLMs deliver routine step instructions reasonably well but fail dramatically at real-time error detection and corrective guidance.

Background & Motivation¶

Multimodal Large Language Models (MLLMs) have made remarkable strides in offline video comprehension and static question answering, yet a massive capability gap remains when attempting to deploy them as real-time procedural coaches. In complex real-world activities such as furniture assembly, daily cooking, mechanical repairs, or fitness training, human performers frequently deviate from the canonical procedure—skipping crucial sub-steps, picking up the wrong tools, or executing actions in an incorrect sequence. A reliable AI assistant cannot remain a passive narrator merely commenting on recorded events; it must operate within strict causal temporal constraints in a closed loop, continuously observing local execution, catching subtle mistakes, and proactively intervening at precise moments to steer the user back onto the correct path.

However, existing procedural activity benchmarks and evaluation protocols cannot support this complete closed-loop cycle. On one hand, conventional procedural video datasets (such as Epic-Kitchens, YouCook2, COIN, and IKEA) predominantly capture expert demonstrations that contain only correct, step-level action annotations, lacking negative supervision signals from user errors and recovery behaviors. On the other hand, although a few error-aware corpora (such as CaptainCook4D, Assembly101, and EgoPER) introduce mistake labels, they stop at post-hoc offline classification without providing actionable corrective guidance, let alone tracking whether subsequent actions recover from errors. Crucially, streaming benchmarks like StreamingBench inject artificial question-answering queries at pre-determined timestamps, merely testing when to answer known questions rather than evaluating whether an agent can independently decide when to remain silent and when to speak without any external prompt.

To overcome these structural limitations, this paper establishes the first multi-domain streaming benchmark for closed-loop procedural assistance. The investigation formalizes task assistance as an instruct-observe-correct cycle and decouples assessment into temporal alignment, intervention behavioral timing, and semantic response quality. Core idea: construct GuideMe, a multi-domain streaming benchmark spanning cooking, object manipulation, daily tasks, and fitness, coupled with a three-component evaluation framework of temporal-semantic bipartite matching, intervention timing classification, and LLM-as-a-Judge, to rigorously audit and constrain the proactive corrective capabilities of MLLMs in continuous causal streams.

Method¶

Overall Architecture¶

GuideMe formulates interactive procedural assistance as an online streaming task under strict causal constraints. At inference time, the assistant receives a single task objective query at the onset of the video; thereafter, it sequentially ingests incoming visual frames via a sliding window with zero access to future frames, deciding autonomously whether to output Silent or issue guidance. The framework is grounded in two primary pillars: a three-stage automated annotation pipeline that synthesizes high-fidelity closed-loop interactions from fragmented procedural datasets, and a three-component evaluation framework that provides threshold-free assessment of temporal alignment, intervention behavior, and content quality.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Source Procedural Videos<br/>EgoPER / CaptainCook4D / HoloAssist / QEVD"] --> B["Three-Stage Automated Annotation Pipeline<br/>Action Extraction & Filtering → Knowledge Induction → Aligned Dialogue Generation"]
    B --> C["Causal Streaming Online Inference<br/>60s Visual Window + Dialogue History → Decide Silent or Intervene"]
    C --> D["Temporal-Semantic Bipartite Matching<br/>Gaussian Decay Temporal Cost + Text Cosine Cost Minimization"]
    C --> E["Intervention Timing Behavioral Classification<br/>CS / FA / NR / PC Quadrant Categorization & Aggregate Score"]
    D --> F["LLM-as-a-Judge Content Quality Scoring<br/>Factual Plausibility & Causal Guidance Grading on Matched Pairs"]

Key Designs¶

1. Three-Stage Automated Annotation Pipeline: Curating Closed-Loop Samples from Heterogeneous Sources

Prior procedural datasets containing mistake records vary drastically in action granularity, state graphs, and corrective textual descriptions, making them incompatible with streaming training. To synthesize scalable, high-quality interaction samples, the authors develop a three-step automated pipeline: - Instructional Activity Extraction and Error Categorization: From EgoPER, CaptainCook4D, HoloAssist, and QEVD, atomic action sequences are extracted. Duplicate consecutive actions are merged and low-information transitional actions (such as brief hand rotation or holding) are pruned. Each atomic action is categorized into a correct action (following canonical steps), a wrong action (deviating from workflow), or a correction action (rectifying an error). - Procedural Knowledge Induction: An LLM infers standard task names and canonical ordered key steps from the filtered action sequences. Because input actions may contain erroneous or corrective steps that could pollute the logical flow, a parallel sampling strategy (\(N=10\)) is deployed, followed by a consensus aggregation step that retains cross-candidate consistent steps and discards contradictions. - Action-Boundary-Aligned Conversation Generation: Videos are partitioned into chunks of up to 200 atomic actions conditioned on global task knowledge and local sub-step contexts. The LLM generates pre-action instructions at action start timestamps, and at action end timestamps produces either execution completion feedback or error detection with corrective guidance. For datasets without explicit correction text, discrepancies between deviated nodes and canonical succeeding nodes in the source task graphs are computed to derive corrective instructions.

2. Temporal-Semantic Bipartite Matching: Sequence-Level Alignment Without Rigid Thresholds

In streaming environments, model-generated utterance timestamps cannot align perfectly frame-by-frame with reference timestamps, rendering conventional rigid metrics like TimeDiff ineffective due to zero tolerance for slight time shifts and lack of semantic awareness. The proposed framework aligns the generated sequence \(G=\{(g_i, t_i)\}_{i=1}^N\) with the reference sequence \(R=\{(r_j, \hat{t}_j)\}_{j=1}^M\) via bipartite matching. The video is partitioned into non-overlapping segments around the midpoints of consecutive reference timestamps, ensuring that each segment contains exactly one active reference event. Within each segment, a cost matrix \(C_{i,j} = \mathcal{L}_{\text{text}}(g_i, r_j) + \mathcal{L}_{\text{dist}}(t_i, \hat{t}_j)\) is computed, where the textual cost is defined as:

\[ \mathcal{L}_{\text{text}}(g_i, r_j) = \begin{cases} 0, & \text{if } g_i, r_j \in \text{Silence} \\ 1 - |\cos(\mathbf{e}_i, \mathbf{\hat{e}}_j)|, & \text{otherwise} \end{cases} \]

with sentence embeddings \(\mathbf{e}_i, \mathbf{\hat{e}}_j\). The temporal distance cost applies a Gaussian decay penalty: \(\mathcal{L}_{\text{dist}}(t_i, \hat{t}_j) = 1 - \exp\left(-\sigma |t_i - \hat{t}_j|^2\right)\). After obtaining the optimal assignment \(\mathcal{A}\) using the Hungarian algorithm, continuous cosine similarities \(S_{i,j} = |\cos(\mathbf{e}_i, \mathbf{\hat{e}}_j)|\) serve as soft weights to calculate soft precision (sPrecision), soft recall (sRecall), and their harmonic mean Soft-F1 (sF1), delivering a unified, threshold-free measure of temporal-semantic coverage.

3. Behavioral Classification and Response Scoring: Balancing Silence and Intervention

In streaming interaction, knowing when to stay silent is just as critical as knowing when to intervene. General coverage metrics cannot distinguish whether a model suffers from excessive chatter (over-intervention) or unresponsiveness (under-intervention). The benchmark constructs two balanced sets of test instances for each video: assistant-anchored instances located at ground-truth intervention timestamps, and silent instances sampled from intervals at least 5 seconds away from any anchor. Each model prediction is classified into four mutually exclusive behavioral outcomes: - Correct Silent (CS): Correctly outputting Silent during a silent instance; - False Alarm (FA): Generating an unnecessary response during a silent instance; - No Response (NR): Failing to respond at a ground-truth intervention anchor (outputting Silent); - Partly Correct (PC): Successfully intervening at a ground-truth intervention anchor.

To summarize response quality, CS receives 100 points, FA and NR receive 0 points, and PC instances receive an LLM-as-a-Judge score scaled between 0 and 100 based on factual plausibility and causal reasoning. The average over all instances forms the aggregate Score, directly penalizing both false alarms and missed interventions.

A Worked Example¶

Consider a microwave mug pizza preparation sequence: - 02:55 - 03:01 Sub-step Warning Point: The user has added herbs and prepares to add salt, but picks up a wet, ingredient-caked measuring spoon. - Ground-Truth (GT): "Hold on a second. It looks like you're about to add the salt, but please make sure you're using a clean, dry measuring spoon so the ingredients don't stick." - Observed Model Responses: Doubao-Seed-1.8 and Qwen3-VL completely miss the spoon hazard and issue standard next-step instructions ("Great, herbs are added. Now add mozzarella cheese"); GPT-5.2 and MMDuet2 output Silent, registering an NR failure. - 05:06 - 05:07 Tool/Ingredient Error Intervention Point: The user reaches for salsa instead of the required marinara sauce for the pizza base. - Ground-Truth (GT): "Just a heads-up: it looks like you're reaching for the sauce, but make sure you use marinara, not salsa." - Observed Model Responses: Most commercial baselines generically say "Great, you grabbed the pizza sauce, add 2 tablespoons", failing to correct the ingredient mismatch and causing downstream procedural failure.

Key Experimental Results¶

Main Results¶

The main benchmark evaluation on GuideMe-Test (473 videos, 46.7 hours, 9,876 interaction samples) includes proprietary MLLMs, open-source general MLLMs, and open-source streaming models using Anchor-Based sampling and a 60-second sliding window.

Model Category	Model	Param.	sPrecision (%) ↑	sRecall (%) ↑	sF1 (%) ↑	\(Score_m\) ↑	CS (%) ↑	NR (%) ↓	FA (%) ↓	PC (%) ↑	Score ↑
Proprietary	Doubao-Seed-1.8	-	30.6	47.5	36.8	63.1	8.8	6.1	34.8	44.2	38.8
	GPT-5.2	-	30.2	36.8	32.5	64.4	13.4	16.8	30.6	39.2	38.9
	Gemini 3 Pro	-	39.3	36.9	36.5	66.6	17.5	22.0	19.0	41.5	44.8
	Gemini 3.1 Pro	-	29.3	46.9	35.7	61.5	7.4	6.3	36.4	49.9	37.8
Open-source	Qwen2.5-7B	7B	30.1	30.9	29.3	57.5	19.7	20.4	25.4	34.5	37.5
	Qwen3-VL-8B	8B	29.7	46.8	35.7	62.0	5.2	6.7	37.7	50.5	31.9
	Qwen3-VL-30B-A3B	30B	29.5	46.9	35.6	61.6	6.7	6.0	36.5	50.9	33.3
	Qwen3.5-397B-A17B	397B	38.4	20.2	21.7	64.6	35.2	37.9	7.9	19.0	45.7
Streaming	VideoLLM-online	8B	22.3	41.1	28.7	40.8	0.0	0.0	43.6	56.4	22.6
	Dispider	7B	1.0	0.1	0.1	46.7	43.6	56.3	0.0	0.1	43.6
	LiveStar	8B	20.7	19.0	19.1	14.4	21.2	24.4	22.2	32.3	25.0
	MMDuet2	3B	3.3	0.2	0.4	43.3	41.7	57.9	0.2	0.2	41.8
Fine-tuned	Qwen3-VL-8B†	8B	40.3	43.7	39.7	58.1	19.5	20.6	24.5	35.3	42.1

Note: † indicates fine-tuning on the GuideMe training split.

Ablation Study¶

1. Breakdown by Ground-Truth Interaction Category (Gemini 3 Pro vs. Doubao-Seed-1.8)

Model	Response Category	sF1 (%) ↑	NR (%) ↓	PC (%) ↑	Score ↑
Gemini 3 Pro	Routine Instruction	42.0	33.3	66.7	33.0
	Error Detection	33.8	39.6	60.4	20.2
	Corrective Guidance	34.3	41.7	58.3	23.1
Doubao-Seed-1.8	Routine Instruction	49.6	8.9	91.1	41.9
	Error Detection	41.9	14.8	85.2	26.2
	Corrective Guidance	42.7	15.0	85.0	30.0

2. Inference Protocol & History Dependency Ablation (100 Sampled Test Videos)

Model	Protocol & Context Setting	sPrecision (%) ↑	sRecall (%) ↑	sF1 (%) ↑	CS (%) ↑	NR (%) ↓	FA (%) ↓	PC (%) ↑	Score ↑
Doubao-Seed-1.8	Dense (1s interval)	23.1	26.3	21.9	89.3	4.4	4.7	1.6	90.3
	Anchor* (Default)	30.6	47.9	36.9	7.8	6.1	34.3	51.9	38.6
	w/o GT (Model-predicted history)	43.5	14.2	20.0	33.9	50.9	8.4	6.8	38.3
Gemini 3 Pro	Dense (1s interval)	10.0	53.2	14.5	45.4	2.1	48.4	4.1	47.9
	Anchor* (Default)	39.7	35.6	35.7	18.1	23.1	19.0	39.9	43.7
	w/o GT (Model-predicted history)	35.7	25.1	28.0	27.6	36.4	14.6	21.3	39.1

Key Findings¶

Bimodal Behavioral Polarization: Evaluated models split sharply into aggressive and conservative camps. Aggressive models (e.g., VideoLLM-online, Qwen3-VL-8B, Doubao) intervene constantly (PC > 50%) but exhibit massive false alarm rates (FA > 34%), essentially failing to remain silent. Conversely, conservative models (e.g., Dispider, MMDuet2, Qwen3.5-397B) collapse into near-total silence, achieving high CS scores solely by missing most necessary interventions (NR between 37% and 57%).
Severe Capability Asymmetry: Across all architectures, performance degrades substantially when shifting from routine step instructions to error detection and corrective guidance. Gemini 3 Pro drops from 33.0 Score on Instruction to 20.2 on Error and 23.1 on Correction. Models can dictate what should happen next, but struggle to verify what is actually happening.
Model Scale and Streaming Pretraining Are Insufficient: Scaling parameters does not resolve streaming decision failures (e.g., Qwen3.5-397B obtains an sF1 of only 21.7, trailing the 8B model). Existing streaming-specific models collapse toward pathological extremes (Dispider achieves an sF1 of 0.1, whereas VideoLLM-online scores 0.0% CS).
Compounding History Degradation: When ground-truth dialogue history is replaced with self-predicted history (w/o GT), missed interventions (NR) surge dramatically (Doubao surges from 6.1% to 50.9%), demonstrating that early streaming mistakes quickly poison downstream temporal context.

Highlights & Insights¶

Establishment of Closed-Loop Streaming Benchmarking: Prior work segregated video reasoning from proactive agent behavior; GuideMe bridges this gap by unifying long-form video, causal streaming constraints, and corrective feedback cycles.
Soft-Weighted Bipartite Matching Paradigm: Employing continuous semantic cosine weights eliminates the fragility of hard-boundary temporal slicing, while the four-quadrant behavioral breakdown isolates over-intervention from under-intervention.
Silence Calibration as the Core Bottleneck: The empirical findings confirm that the primary obstacle in streaming assistants is not sentence generation, but calibrating when to withhold speech, providing clear motivation for future reinforcement learning with silence penalties.

Limitations & Future Work¶

Open-Loop Evaluation of Execution Dynamics: Because the benchmark evaluates pre-recorded videos, an assistant's corrective advice cannot dynamically alter the human actor's subsequent physical trajectory in real time.
Trade-offs in Streaming Sampling Granularity: Full 1-second dense sampling incurs extreme computational and API costs; hence Anchor-Based sampling is adopted as a pragmatic proxy.
Integration with Interactive Embodied Simulators: Future iterations could extend these procedural coaching tasks into interactive 3D simulators or human-in-the-loop environments where corrective guidance branches into different execution outcomes.

vs HoloAssist / QEVD: While HoloAssist and QEVD target specific domains (assembly and fitness) with offline error annotations or fixed feedback, GuideMe spans four major domains and models the full closed loop encompassing instructions, feedback, error detection, and actionable correction.
vs VideoLLM-online / StreamingBench: Existing streaming benchmarks rely primarily on pre-scheduled external queries (effectively streaming video QA). GuideMe mandates that the model autonomously monitor visual streams and determine intervention timing without external triggers.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First multi-domain streaming benchmark for closed-loop procedural task guidance and intervention.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates proprietary, open-source, and streaming models with comprehensive behavioral classification and inference ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear conceptual formulation, rigorous mathematical metric definitions, and high-quality presentation.
Value: ⭐⭐⭐⭐⭐ Identifies the critical failure modes preventing MLLMs from functioning as real-world real-time procedural coaches.