Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos¶
Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: Video Understanding / Agent
Keywords: proactive assistant; egocentric video; streaming memory; retrieval-augmented reasoning; proactivity benchmark
TL;DR¶
Vinci2 reframes "should an egocentric assistant speak up on its own?" as a decision problem over long-horizon context, and delivers both EgoServe — the first benchmark for proactive assistance in continuous egocentric video (3,000+ service instances, 4 temporal memory horizons, 10 subcategories) — and EgoMemo, a training-free memory-augmented agent (multi-scale temporal summaries + an evolving knowledge graph + a visual embedding archive, with three parallel retrieval pathways and VLM caption reconstruction), which lifts overall F1 on EgoServe from 4.7 (GPT-5-mini baseline) to 8.0 while remaining competitive on five existing benchmarks.
Background & Motivation¶
Egocentric assistants today operate under essentially two paradigms. The first is reactive: the vast majority of Video-LLMs (VideoChat, Apollo, etc.) answer only after an explicit user query and expose no interface for initiating interaction. The second is semi-proactive: event-triggered systems such as StreamBridge, EWO and ProAssist require the user to supply a task instruction up front, then monitor the stream for predefined events and respond on detection; the on-device assistant Vinci belongs to this line as well. Both rest on an unexamined assumption — that once something has been detected, something ought to be said.
The real difficulty is that semi-proactive systems are confined to the scope of the upfront instruction and to the immediate visual context. They cannot draw on observations accumulated over minutes, hours or days, and they have no mechanism for assessing whether interrupting the user right now is actually worth it. This is precisely where proactive assistance parts ways with event detection: a badly timed reminder costs more than one wasted generation — it spends the user's trust — so "when to stay silent" and "when to speak" must be two faces of a single decision, and that decision can only be grounded in the user's own history, habits and current activity. The same observation ("the user picked up a phone") is an interruption for someone cooking and corroborating evidence for someone who spent yesterday repeatedly recording with a phone timer.
This paper's angle is to first make the problem measurable, then demonstrate feasibility. The authors formalize proactive assistance as a joint decision-and-generation task over streaming egocentric perception: at each timestep the agent observes the current clip and emits a binary intervention decision plus, when the decision is 1, a service response; a correct response must simultaneously satisfy a temporal window, the service category, and content relevance. To make this definition evaluable, the paper builds the EgoServe benchmark, which organizes proactive services into four temporal memory horizons — Instant, Short-Term, Episodic and Long-Term — according to how much history each requires, and then proposes the training-free EgoMemo as a baseline system on top of it. Core idea: elevate "should I intervene?" from a by-product of event detection into an explicit reasoning output — the agent first judges whether the current observation warrants intervention, and if deeper context is needed it retrieves in parallel over temporal, semantic and visual memory, uses a VLM to reconstruct the retrieved evidence into a coherent narrative, and lets the same reasoner emit both the intervention decision and the service content.
Method¶
Overall Architecture¶
EgoMemo takes a continuous egocentric video stream \(V=\{v_1,v_2,\dots\}\) as input and outputs, at each timestep, an intervention decision \(d_t\in\{0,1\}\) together with a response \(r_t\). The pipeline has two stages, and both are incremental — neither ever requires access to the complete video. Stage one, streaming memory construction, segments the stream into non-overlapping short clips, generates a timestamped dense caption and sampled keyframes for each, and grows three complementary memory representations incrementally from those atomic inputs. Stage two, streaming retrieval-augmented reasoning, receives the current clip caption plus the most recent short-term context at every timestep, decides whether deeper historical evidence is needed, generates a retrieval query if so, invokes three parallel pathways, and hands the synthesized context to the reasoner for the final decision.
The same architecture serves two modes with no structural modification. In proactive mode the trigger is the agent's own judgment about the current observation, so it is free to emit \(d_t=0\) and say nothing; in reactive mode the user query \(q\) is an external trigger with \(d_t=1\) by default, and the identical retrieval-and-reasoning pipeline runs. This unification is deliberate: proactivity is not a bolt-on switch but the natural outcome of letting the retrieve-then-reason backbone keep running when no external trigger arrives.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Egocentric video stream<br/>clip + caption + keyframes"] --> B["Multi-scale temporal memory"]
A --> C["Evolving knowledge graph"]
A --> D["Visual embedding archive"]
A -->|current observation suffices| F["Intervention decision & response<br/>d_t=0 stay silent / d_t=1 serve"]
B --> E["Parallel retrieval & caption reconstruction"]
C --> E
D --> E
E --> F
Key Designs¶
1. Multi-scale temporal memory: retrieval at the right temporal granularity
Captions are EgoMemo's only textual source of fact, but a single clip caption is only meaningful at the scale of seconds — using it to answer "what are this user's daily routines" fails immediately, while keeping only hour-level summaries discards exactly the detail a safety alert depends on (whether the knife is held by the handle or grabbed by the blade). The paper therefore organizes captions into a three-level hierarchy: clip-level entries preserve fine-grained perceptual detail, activity-level summaries periodically aggregate consecutive clip captions to characterize one stretch of activity, and session-level summaries aggregate activity entries to encode long-horizon routines. The roll-up itself is a plain hierarchical summarization, \(M_A^{(j)}=\text{Summarize}(\{c_t\}_{t\in\mathcal{W}_j})\), with the session level aggregating activity entries the same way (the rendering of the original equation is corrupted; this restores it by semantics, ⚠️ refer to the original paper). The key property is that the roll-up is fully incremental: only newly accumulated segments trigger summarization at the next level, and completed levels are never recomputed. All three levels are then encoded into dense vectors by one text encoder and indexed for similarity search, so "did the user do this yesterday" can be asked at the session level and "was that step a mistake" at the clip level, without forcing both questions into a single embedding space. In the ablation, collapsing the hierarchy to a single scale (w/o MS) drops overall F1 from 8.0 to 7.0, with Memory Link falling from 4.9 to 1.9 and Routine Optimization from 11.8 to 5.7 — the hierarchy is a capability prerequisite, not a storage optimization.
2. Evolving knowledge graph: making cross-segment entity associations explicit
Pure vector retrieval has a structural blind spot: it returns clips by lexical or semantic similarity, which makes "is this the same thing I encountered last week?" very hard to answer. Users may name the same object differently on different days, and descriptions can share no vocabulary at all once the scene changes — precisely the associations that Long-Term services (Memory Link, Habit Coaching) depend on. EgoMemo maintains a knowledge graph \(G=(N,E)\) that evolves as observations arrive: for each new clip caption an LLM extracts entities and relations, which are then merged into the global graph through name-based entity resolution, so that one real-world entity becomes one node that keeps accumulating links to its source captions. The graph thus becomes a structured index layered over temporal memory: when a retrieval query hits a node, the system can expand to its one-hop neighbors and pull in events that are semantically related but were never mentioned in the same sentence. Removing the graph pathway (w/o GSR) lowers overall F1 to 6.5 and zeroes Memory Link — the only single-component removal in the whole ablation that does so, showing that cross-time entity linking genuinely cannot be compensated for by temporal or visual retrieval.
3. Visual embedding archive: retrieving what resists verbalization
A caption is a lossy compression of the frame, and visual details that are hard to put into words — object appearance, spatial layout — tend to be dropped at captioning time, even though they are exactly what supports judgments like "the same mug" or "the same corner of the room." EgoMemo therefore keeps a visual archive as well: sampled keyframes from each clip are encoded by a multimodal encoder and stored alongside their source caption index and timestamp. Because the query is textual, the system first rewrites it into a visual-centric description emphasizing visual attributes, encodes that with the same multimodal encoder, and runs top-k similarity search over the archive, mapping the retrieved keyframes back to their source clip captions. The value of this pathway is that it can recall moments with no lexical overlap with the query — the user says "my keys" while the archive holds a caption reading only "a small shiny metal object on the table." Disabling it (w/o VSR) gives 8.0 → 6.9, the smallest drop of the three pathways, but the effect is concentrated and highly interpretable: Memory Link 4.9 → 3.2 and Routine Optimization 11.8 → 6.4, the two services that need recurring objects and scenes to confirm "this is the same thing."
4. Parallel retrieval and VLM caption reconstruction: turning heterogeneous evidence into a reasoner-ready context
The first three designs decide what to remember; this one decides how to use it. At each timestep the reasoner receives the current clip caption and the recent short-term context and judges whether deeper historical evidence is warranted; if so, it generates a retrieval query \(q_r\) and fires three pathways at once. Temporal retrieval takes the top-k similar entries over the three-level summary embeddings (constrained to \(t'\le t\) to preserve the streaming setting, and free to hit any of the clip, activity and session levels so that questions at different granularities are served). Graph-based semantic retrieval matches entities in \(q_r\) against graph nodes and expands to one-hop neighbors. Visual similarity retrieval follows the archive pathway described above. All three return caption indices rather than self-contained descriptions: the graph and visual routes hand back little more than "which captions are relevant," and dumping those scattered clips into an LLM leaves it unable to resolve cross-clip co-reference and still missing the visual detail destroyed at captioning time. This is why the w/o Recons. ablation zeroes both Memory Recall and Memory Link and drops overall F1 to 6.6, the largest single drop in the table. The paper's remedy is a unified VLM caption reconstruction step: for each retrieved index set, the corresponding clip captions and keyframes are gathered and a VLM regenerates a reconstructed caption conditioned on the retrieval query, resolving co-reference, recovering visual information absent from the original captions, and organizing the result into a query-focused narrative. The directly retrieved temporal captions, the graph-reconstructed context and the visually reconstructed context are merged into a unified retrieved context \(\hat{\mathcal{C}}\) and passed to the reasoner together with the current caption and query to produce \(d_t\) and \(r_t\).
One boundary worth stating: the paper never gives an explicit threshold formula or scoring function for "should I help right now." Whether to trigger is the reasoner's own prompt-driven judgment, and retrieval is only invoked when deeper evidence is deemed necessary (prompt and hyper-parameter details live in the supplementary material, ⚠️ refer to the original paper). EgoMemo's restraint therefore comes from the language model's judgment rather than from a tunable criterion — which is exactly the gap the authors flag for future work, listing "learning intervention timing from human preferences" as a next step.
A Worked Example¶
The right-hand example of Fig. 4 walks the whole pipeline once. On Day 2 the user raises a phone with a timer on screen; on its own, the current clip caption is unremarkable — nothing in that single moment is worth alerting anyone about. The reasoner nevertheless judges that this observation needs deeper context and generates a retrieval query, firing all three pathways. Clip-level temporal retrieval recovers several similar "raising the phone" moments from earlier that day, while the session-level summary supplies "this user has a habit of recording their own process at length." Graph retrieval expands the node for "phone" one hop and surfaces associated nodes that co-occurred earlier, such as a stand or a particular camera angle. The visual archive recalls similar frames of a lit phone screen from several previous days, some of whose captions never mention a phone or a timer at all. The three index sets are each reconstructed into a caption by the VLM and merged into a cross-session account of the activity pattern, on which basis the reasoner classifies this as a Routine Optimization service — it does not say "you are looking at your phone," it suggests how to make the recording routine less cumbersome, grounded in several days of repeated observation. The contrasting trigger from the same model is pure instant reaction: observing the user scrubbing a knife barehanded immediately fires a Safety Alert from the current clip caption alone, with no retrieval at all. Side by side, the two examples show how one pipeline stretches on demand within the same moment — Instant services take the zero-retrieval direct route while Long-Term services take the three-pathway retrieval plus reconstruction route.
Loss & Training¶
EgoMemo is entirely training-free; every capability is assembled from off-the-shelf models. Clip captions come from Qwen3-VL-8B-Instruct, activity- and session-level summaries are produced by LLM summarization, entity and relation extraction for the knowledge graph uses GPT-4o-mini, the text encoder is OpenAI text-embedding-3-small, keyframes are encoded with ImageBind, and the reasoning agent is GPT-5-mini on streaming benchmarks and GPT-5.2 on offline benchmarks (⚠️ the main text says GPT-5.2 while the corresponding reference entry is labeled GPT-5.1; refer to the original paper). Temporal windows adapt to video length: ultra-long recordings such as EgoLife use 30s / 5min / 1h for the clip / activity / session levels, while shorter online benchmarks use 10s / 1min / 5min. The temporal tolerance window \(\delta\) used at evaluation time is likewise set per dataset according to its characteristic timescale: 60s for EgoLife, 25s for CaptainCook4D and 10s for HoloAssist. Notably, the paper also explains how the same architecture transfers to offline tasks: build the complete memory in a single forward pass over the whole video, then answer with a coarse-to-fine strategy that first tries the activity- and session-level summaries alone and only falls back to clip-level retrieval when the agent judges the available information insufficient — so the offline adaptation, too, requires no architectural change.
Key Experimental Results¶
Main Results¶
Overall and per-subcategory F1 on EgoServe (Instant covers SA/TU, Short-Term covers NSG/ER/RR, Episodic covers MR/TR, Long-Term covers HC/ML/RO; LLM-score is the mean 1–5 rating of responses on matched prediction–ground-truth pairs):
| Model | SA | TU | NSG | ER | RR | MR | TR | HC | ML | RO | Overall F1 | LLM-score |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-VL-Plus | 5.2 | 1.5 | 8.6 | 10.4 | 3.4 | 0.0 | 1.8 | 4.4 | 0.0 | 0.0 | 3.5 | 2.8 |
| GPT-5-mini | 12.5 | 3.6 | 9.5 | 1.0 | 5.2 | 0.0 | 9.4 | 5.7 | 0.0 | 0.0 | 4.7 | 3.0 |
| EgoMemo (Ours) | 11.4 | 7.5 | 24.7 | 1.7 | 4.7 | 3.8 | 5.7 | 3.7 | 4.9 | 11.8 | 8.0 | 2.8 |
Generalization to existing benchmarks (EgoMemo is training-free throughout; EyeWO is the only trained model in the ESTP-Bench comparison):
| Benchmark | Metric | EgoMemo | Best prior | Note |
|---|---|---|---|---|
| ESTP-Bench (explicit proactive) | average | 27.6 | 23.6 (EyeWO) | TRU 32.4 vs 25.1, EOSC 35.8 vs 20.8 |
| ESTP-Bench (implicit proactive) | average | 34.7 | 52.5 (EyeWO) | the only trained model leads; all others are training-free |
| OVO-Bench (real-time perception) | average | 75.15 | 64.46 (GPT-4o) | backward tracing 49.60 vs GPT-4o 60.75 |
| EgoSchema | accuracy | 74.8 | 67.6 (EgoThinker) | +7.2 |
| QAEgo4D | accuracy | 68.0 | 66.2 (EgoThinker) | long-form episodic memory querying |
| EgoTaskQA | accuracy | 60.9 | 64.4 (EgoThinker) | slightly behind on fine-grained state reasoning in short procedural segments |
Ablation Study¶
Component-wise removal on EgoServe (overall F1 together with the most affected subcategories):
| Config | Overall F1 | Key change | Note |
|---|---|---|---|
| EgoMemo | 8.0 | — | full model |
| w/o MS | 7.0 | ML 4.9→1.9, RO 11.8→5.7 | three-level temporal hierarchy collapsed to one scale |
| w/o Recons. | 6.6 | MR 3.8→0.0, ML 4.9→0.0 | VLM caption reconstruction removed (largest drop) |
| w/o VSR | 6.9 | ML 4.9→3.2, RO 11.8→6.4 | visual similarity retrieval removed |
| w/o GSR | 6.5 | ML 4.9→0.0 | graph-based semantic retrieval removed (only removal that zeroes ML) |
| w/o MTR | 6.8 | ML 4.9→2.1 | multi-scale temporal retrieval removed |
Key Findings¶
- The baselines' bottleneck is when to speak, not what to say. GPT-5-mini matches far fewer service instances than EgoMemo (overall 4.7 vs 8.0), yet the LLM-scores are nearly identical (3.0 vs 2.8, with Qwen also at 2.8) — once a prediction lands in the right category and temporal window, response quality is not where the gap lies; picking the moment is.
- Services with longer histories separate the methods. Both baselines score 0 on the long-term ML and RO categories while EgoMemo reaches 4.9 and 11.8; Qwen3-VL-Plus scores 0 even on the episodic MR. This matches the paper's design assumption that the capability gap is driven mainly by access to accumulated context.
- The three retrieval pathways are complementary; none can be dropped. Removing any one costs 6.5–6.9 overall, and the damage lands in different places: the graph matters for Memory Link, the visual pathway for Memory Link and Routine Optimization, and temporal retrieval has the broadest reach (ML falls to 2.1).
- Reconstruction outweighs any single retrieval pathway. The 8.0 → 6.6 drop for w/o Recons. is the largest single-point loss, confirming the judgment that "retrieved indices are not the same as reasoner-ready evidence": a structured index must be re-narrated under the query before the reasoner can use it.
- Absolute scores are uniformly low (the best method reaches only 8.0 overall and 2.8/5 on LLM-score), which the authors read as task difficulty rather than method failure — temporal window, category and content must all line up at once, and EgoServe is positioned as a diagnostic tool for exposing exactly these challenges.
Highlights & Insights¶
- The cost of interrupting is written into the task definition. Correctness requires the temporal window, the service category and content relevance to hold simultaneously, so a "say less, err less" strategy is penalized just as hard through recall — the benchmark refuses to reward a silent assistant, which is what sets it apart from event-triggered evaluation.
- Memory is layered by temporal horizon, not by modality. The four service categories (Instant / Short-Term / Episodic / Long-Term) are defined by how much context they require, so one memory structure directly answers "how much history does this service need" and explains why a single-granularity store errs in two directions at once: too short false-alarms, too long misses.
- The three retrieval pathways divide labor orthogonally rather than redundantly. Temporal retrieval handles "when," the graph handles "is it the same thing," and the visual archive handles "can it be put into words"; their failure modes do not overlap (their ablation drops land in different categories). This heterogeneous-parallel-retrieval template transfers to any long-horizon memory agent, especially systems that already have a vector store but no entity-level linkage.
- A streaming-first design pays off offline. Because memory construction was incremental to begin with, the offline setting needs only one forward pass to build the stores plus a coarse-to-fine access policy (summaries first, retrieval on demand), and it beats the dedicated EgoThinker by 7.2 points on EgoSchema. The "streaming first, offline after" ordering deserves imitation in other long-video work: it forces the system to depend only on the past at every step, which structurally rules out the future-information leakage that plagues offline evaluation.
- The engineering value of training-free. The whole pipeline is assembled from off-the-shelf models with no parameter updates, which means the benchmark ships with a reproducible strong baseline and indicates that the current bottleneck lies in how memory is organized rather than in model capability.
Limitations & Future Work¶
The authors acknowledge three limitations: captioning loses fine-grained visual detail (text memory is the primary carrier), name-based entity resolution can fail for visually ambiguous entities, and the semi-automated annotation pipeline may bias the benchmark toward service types that foundation models find easier to generate.
The evaluation design raises several further questions. First, the temporal tolerance window \(\delta\) is set by hand per dataset (EgoLife 60s / CaptainCook4D 25s / HoloAssist 10s), so F1 values are not directly comparable across sources — yet the Overall column averages exactly these heterogeneous numbers, and readers should keep its aggregate meaning in mind. Second, coverage across the three sources is uneven: HoloAssist annotations reach only Instant and Short-Term by construction and CaptainCook4D concentrates on error correction, so long-term services rest almost entirely on EgoLife and the Long-Term conclusions stand on a single data source. Third, the LLM-score of only 2.8/5 means response quality itself stays mid-to-low even when predictions are correctly matched, and since that score is computed by an automatic judge over matched pairs only, it is subtly coupled to recall in a way that can flatter methods matching fewer instances (the baselines' 3.0 versus EgoMemo's 2.8 in Table 1 is an instance of this coupling).
On improvements, the authors' three directions are concrete: learn intervention timing from human preferences instead of relying on prompt-driven judgment, incorporate audio context (speech and sound in egocentric video are often the cues that trigger a service), and extend EgoServe to multi-turn proactive dialogue and multi-user settings. Two more suggest themselves: upgrade entity resolution from name matching to joint visual-textual matching, which the visual archive can already support, and replace the hand-set \(\delta\) with a soft window that adapts per service category so that temporal precision becomes an analysable quantity rather than an evaluation setting.
Related Work & Insights¶
- vs StreamBridge / EWO / ProAssist (event-triggered semi-proactive systems): they are driven by a task instruction the user supplies in advance and respond once a predefined event fires, conflating detection with the decision to intervene and reasoning only over the immediate visual context. EgoMemo differs on two counts: intervention is an explicit reasoning output (it can emit \(d_t=0\)), and the decision rests on retrieved context spanning minutes to days. The trade-off is that EgoMemo's triggering relies on LLM prompt judgment rather than the interpretable, debuggable event definitions those systems enjoy.
- vs MovieChat / MA-LMM / VideoAgent (memory-augmented long-video models): these also address video outgrowing the context window, but their memory is written by a fixed compression policy and inference is assumed to have the full video available. EgoMemo both builds and retrieves memory in a streaming fashion (constrained by \(t'\le t\)) and additionally requires that memory be re-narrated for the retrieval query.
- vs VideoRAG / Vgent / WorldMM (graph-indexed RAG for long video): they likewise introduce graph-driven indexing and multi-type memory, but assume offline access to the whole video and typically feed retrieved results straight to the LLM. EgoMemo differs in being fully streaming and in using VLM caption reconstruction to bridge the information gap between structured retrieval results and the contextual description a reasoner needs — the single largest contributor in the ablation.
- vs Vinci (the prior on-device egocentric assistant): Vinci is a reactive real-time assistant; Vinci2's stated goal is to advance the same line toward proactivity, adding cross-session memory and intervention decisions on the capability side and EgoServe on the evaluation side.
- vs ContextAgent / SensibleAgent / ProAgentBench (proactive assistants and benchmarks): these target language-only or short-horizon sensory contexts (smart glasses, earphones, wearables) and do not involve long-horizon memory or temporal reasoning over continuous egocentric video; EgoServe fills exactly the "proactivity over a continuous video stream" cell.
Rating¶
- Novelty: ⭐⭐⭐⭐ Reframes proactive assistance from event detection into a long-horizon intervention-decision problem and supplies the first matching benchmark; the problem formulation is the main contribution
- Experimental Thoroughness: ⭐⭐⭐⭐ One new benchmark plus five existing ones and six ablation configurations give broad coverage, but the best method reaches only 8.0 overall on the headline benchmark and the Long-Term conclusions rest on a single data source
- Writing Quality: ⭐⭐⭐⭐ The three-paradigm framing is crisp and figures and motivation line up with the method; Table 1 mixes baselines with ablations and some equations are corrupted, so it must be read against the text
- Value: ⭐⭐⭐⭐ The benchmark plus training-free baseline pairing makes "when to interrupt" a measurable, reproducible question for the first time, with direct relevance to both wearable assistants and long-horizon memory agents