Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding

Conference: ECCV 2026
Paper: Official Paper Page
PDF: Full Paper
Code: https://air-embodied-brain.github.io/Em-Garde/
Area: Video Understanding / Proactive Streaming Video Understanding
Keywords: Proactive Response, Visual Proposals, Streaming Matching, Multimodal Embeddings, Trigger Decision-making

TL;DR

Em-Garde reformulates the complex "whether to answer" reasoning performed at every frame into a one-time visual proposal generation during query time and lightweight in-stream matching. It achieves an average F1 of 30.99 on OVO-Bench, 38.0% proactive response accuracy on StreamingBench, and maintains constant-length streaming computation at approximately 13 fps.

Background & Motivation

Proactive streaming video assistants do not wait for users to ask questions after seeing an event; instead, they first receive instructions, then continuously observe infinite video streams, and proactively speak when target events occur. For example, if a user says "remind me when the water boils", the system must determine at each subsequent moment whether the current frame satisfies the instruction. Methods like VideoLLM-Online and Dispider model this as a frame-by-frame answer/silence decision, while schemes like OVO-Bench repeatedly query a VideoLLM to determine if the target event has occurred. Regardless of the form, fine-grained visual recognition, event transition detection, and instruction relevance judgment are stuffed into high-frequency streaming loops.

The issue is that the streaming loop must keep up with at least 5--10 frames per second, while deep visual semantic reasoning relies on larger models and rich context. Shrinking the model or aggressively compressing history can speed up inference but hurts visual grain and trigger quality. While using long-context planning or reinforcement learning decisions is more accurate, the computational cost grows with the video length. What is truly redundant here is not the visual observation itself, but the semantic explanation of the same user instruction: the system re-understands "what situation is worth answering" at every frame.

Em-Garde observes that when a query arrives, "situations worth answering" can first be translated into evidence identifiable in short videos (e.g., expanding "water boiling" to "steam continuously rising" or "vigorous bubbling on the water surface"). Subsequent high-frequency loops only need to detect this evidence, without re-interpreting the instruction from scratch. Core Idea: Decompose proactive triggering into instruction-guided visual proposal generation at query time and lightweight proposal matching during the streaming phase, moving expensive semantic reasoning out of the high-frequency loop.

Method

Overall Architecture

At the query moment, the system receives the instruction and the video history of the past 5 seconds. The Instruction-Guided Proposal Parser (IGPP) invokes Qwen2.5-VL-7B to translate the abstract instruction into a set of visual cues locatable in short videos. Upon entering the streaming phase, the Lightweight Proposal Matching Module (LPMM) continuously encodes the most recent 2-second video window and calculates similarity scores with each pre-encoded proposal. When a sufficiently large rise in the similarity of any proposal occurs, the system triggers the downstream Multimodal Large Language Model (MLLM) to generate and verify the response.

This pipeline separates low-frequency but expensive semantic parsing, high-frequency but cheap visual matching, and low-frequency answer generation. The streaming loop only processes a constant-length window, and visual encodings of overlapping frames can be reused, ensuring that the single-step inference time does not increase linearly as the video lengthens.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
        A["User Instruction + Pre-query Video"] --> B["Instruction-Guided Proposal Parsing<br/>IGPP"]
        B --> C["Locatable Visual Proposals"]
        C --> D["Lightweight Proposal Matching<br/>LPMM"]
        V["Streaming Short Video Window"] --> D
        D --> E["Similarity-Rise Triggering<br/>& Encoding Cache"]
        E --> F["Answer Model Generation & Checking"]

Key Designs

1. Instruction-Guided Proposal Parsing: Translating answer conditions into evidence visible in short windows

The task of IGPP is not to directly predict the trigger time, but to define subsequent perception goals. It receives the instruction and the video history around the query, outputting multiple short declarative proposals. A valid proposal must satisfy three criteria: it only appears when an answer is indeed required, it can be matched using only short video segments, and it allows multiple complementary cues to cover the uncertain appearance of the same event. Therefore, facing "what magazine was taken from the shelf", the parser can describe both "hand holding a magazine with a colorful cover" and "a man leaving the shelf holding a magazine" simultaneously, without betting on a single description.

The authors curated Parse2Prop-1K for this reason: the data actually contains 668 queries and proactive responses across 92 COIN, Ego4D, and BEHAVIOR videos, where about half of the samples feature proposal annotations written by humans or GPT-5; each query has an average of 3.99 visual cues, with an average of 13.5 words each. Supervised fine-tuning (SFT) is first conducted to learn the proposal format and preset parsing styles, followed by reinforcement learning using downstream triggering outcomes, allowing the proposals to gradually adapt to the actual perception model rather than just being linguistically plausible.

2. Lightweight Proposal Matching: Replacing frame-by-frame semantic decisions with similarity variations

LPMM utilizes the training-free Ops-MM-V1-2B multimodal embedding model. Each text proposal is encoded only once. During the streaming phase, video representations are generated with a 2-second, 2 fps window and cosine similarity is computed with each proposal. The score of the \(i\)-th proposal at time \(t\) is:

\[ s_i^t=\cos\left(e_v^t,e_{p_i}\right). \]

The system does not set a threshold directly on the absolute similarity, but instead observes the score increment between adjacent moments; as long as any proposal exhibits a jump exceeding threshold \(\theta\), an answer is triggered:

\[ D_t=\mathbb{1}\!\left[\max_i\left(s_i^t-s_i^{t-1}\right)>\theta\right]. \]

Change increments are more effective than absolute values in canceling out baseline similarity differences across different videos. \(\theta\) also becomes a clear deployment knob: lowering it increases recall but raises false triggers and downstream verification efforts, while raising it makes the system more conservative but potentially misses events. The paper sets the default to 0.04 and verifies that the 0.03--0.04 train/test combinations still maintain robust performance.

3. Similarity-Rise Triggering and Encoding Cache: Constant-scale single-step overhead for infinite videos

Proposal generation and final answering can be executed asynchronously at a low frequency; only the short-window embedding matching runs continuously. Adjoining sliding windows heavily overlap, so the authors cache existing frame encodings to only encode one new frame at each tick; this optimization speeds up the visual encoder by an additional 2--3 times. Because the input window is fixed, there is no growing KV cache, nor is there a need to sacrifice historical semantics with high compression rates. The maximum throughput on an A100 is 10--15 fps, with the actual comparison point in performance-throughput diagrams being around 13 fps.

The boundaries of this design are also clear: LPMM is only responsible for detecting "whether the current short segment resembles a proposal" and does not perform long-range reasoning; answer generation is still handed over to a pluggable downstream MLLM to generate and verify based on the instruction and video history up to the trigger point. Modularity reduces triggering overhead without claiming to replace full video QA models with a 2B embedding model.

Loss & Training

IGPP is initialized with Qwen2.5-VL-7B, undergoing ~1 hour of SFT followed by ~5 hours of GRPO on 8 A100 GPUs. Reinforcement learning rewards are determined by matching predicted triggers within a 4-second tolerance window after the actual event onset: hits within the window are counted as correct triggers, while responses outside the window are counted as false positives. Letting \(n_c\) be the number of correct triggers, \(n_{fp}\) the number of false positives, and \(n\) the number of ground truth events, the false positive penalty and total reward are formulated as:

\[ r_{fp}=1-2^{-n_{fp}/n},\qquad r=\frac{(1-\lambda r_{fp})n_c}{n}. \]

\(\lambda\) controls the trade-off between event recall and false positive suppression. The default is \(\lambda=1\); ablation shows that omitting the false positive penalty drops OVO-Bench F1 from 31.0 to 27.7, while an excessively high \(\lambda=1.5\) drops it to 19.3, indicating that proposals must both cover target events and avoid becoming overly broad descriptions that match everywhere. LPMM itself is training-free. During deployment, IGPP observes 5-second, 1 fps history, and LPMM processes the stream sequentially with a 2-second, 2 fps window.

Key Experimental Results

Main Results

Benchmark & Metric	Representative Previous Best	Previous Result	Em-Garde	Gain
OVO-Bench FAR, Average F1	MMDuet-2	20.51	30.99	+10.48
StreamingBench PO, Accuracy	MMDuet-2 (1 fps)	34.6	38.0 (2 fps)	+3.4
OVO-Bench Real-time VP	StreamAgent	61.3	63.0	+1.7
OVO-Bench Backward Tracing	StreamForest	52.0	52.2	+0.2

OVO-Bench FAR also reports online recall and precision across three task categories: CRR, SSR, and REC; Em-Garde achieves corresponding F1 scores of 26.40, 23.40, and 43.16, averaging 30.99. The 38.0 on StreamingBench is the accuracy for the proactive response PO task, which should not be conflated with its online comprehension task of 76.7. Results on ProactiveVideoQA are more mixed: Em-Garde's PAUC on EGO is 52.3, higher than MMDuet-2's 33.6; but on WEB it is 44.3, lower than 53.3; and on VAD it is 27.4, lower than 28.9. Consequently, the paper only claims "competitive" performance rather than all-around state-of-the-art leadership.

Ablation Study

Configuration	OVO-Bench FAR F1	StreamingBench PO Accuracy	Implication relative to the full model
Em-Garde Full Model	31.0	38.0	IGPP + LPMM
w/o IGPP	24.0	28.8	Directly rewrite prompts using Qwen2.5-VL-7B
w/o LPMM	14.6	23.2	Perform frame-by-frame Yes/No decisions using Qwen2-VL-2B
w/o both IGPP and LPMM	18.6	17.2	No longer employ the propose-match decomposition
Naive Sliding Window	21.0	26.8	Same backbone directly observing query and 2-second window
SFT-only proposer	23.4	34.8	Proposals not aligned using trigger feedback
GRPO, \(\lambda=1\)	31.0	38.0	Moderate false trigger penalty

Key Findings

• LPMM is not an optional speed-up helper: removing it drops the F1 on OVO-Bench from 31.0 to 14.6, which is a steeper drop than the 7.0-point decrease when removing the IGPP. This suggests that specialized embedding matching is far more suited for high-frequency triggering than relying on equal-scale VLMs to answer Yes/No per window.
• The naive sliding window using the same backbone only yields 21.0 / 26.8, which rules out the explanation that benefits stem solely from pre-training capacity; what is truly effective is the task reformulation of "propose first, match later".
• When the threshold \(\theta\) is set to 0.04 for both training and testing, OVO-Bench achieves 30.99; the tested 0.03--0.04 combinations still yield at least 26.59. However, applications with different security budgets should re-select thresholds rather than directly copying default settings.
• Online understanding results reach 76.7 on StreamingBench, 63.0 on OVO-Bench Real-time VP, and 52.2 on Backward Tracing, showing that the triggering framework largely preserves the QA capability of the downstream 7B response model; these figures do not imply that the 2B LPMM independently solves long-range QA.

Highlights & Insights

• The most crucial innovation is not making the per-frame model smaller, but changing when the computation occurs. For the same instruction, pre-compiling stable semantic conditions into visual proposals aligns better with the load structure of streaming systems than repeated natural language reasoning at every frame.
• Triggering on similarity "rise" rather than absolute similarity is a highly transferable trick. It shifts focus from "how similar is the current frame to the text" to "did the target evidence just appear", thus mitigating constant high scores caused by the persistent presence of relevant objects in the scene.
• The scale of Parse2Prop-1K is only 668 queries, but downstream triggering rewards allow the parser to learn expressions that the perception model actually recognizes. This suggests that modular interfaces should not be defined solely by linguistic supervision but can also be backward-calibrated based on the success of the receiving module.
• Fixed short windows combined with overlapping encoding caches decouple single-step costs from session length. This makes latency and memory overhead far more predictable than continuously compressing increasingly longer histories, a core system design transferable to anomaly detection, wearable reminders, and robotic event monitoring.

Limitations & Future Work

• The authors explicitly note that the similarity-rise threshold can be sensitive to streaming video uncertainties. Sudden cuts or visually relevant but semantically irrelevant transitions can cause score spikes; negative proposal experiments are limited by existing embedding models and are not yet robust enough to distinguish subtle textual differences.
• The method focuses on "when to answer" and does not solve long-range online understanding. Short windows cannot independently resolve conditions that require very long context histories; downstream answer quality still relies on the base model's capacity to store and retrieve history.
• Proposal parsing relies on a 7B MLLM. Although it only runs at query time and behaves asynchronously, multi-user, high-frequency instruction rewrites or edge deployment must still pay for its first-turn latency and compute costs; the paper does not provide end-to-end latency or power consumption on edge devices.
• Parse2Prop-1K covers 92 videos, and about half of the proposal annotations are provided by humans or GPT-5. For high-risk fields like healthcare or surveillance, the visual representation of open-world events and false-trigger costs may vary wildly from the three training domains, requiring domain-specific data and cost-sensitive calibration.
• More robust future directions include learned temporal change-point detection, multi-proposal joint evidence, explicit negative evidence modeling, and joint optimization of triggering and answer generation. Safety-critical scenarios should also report cost-weighted missing/false alarm rates instead of just average F1.

Related Work & Insights

• vs VideoLLM-Online / Dispider: These methods output silence or answer decisions at every step, with Dispider reducing loads by separating perception, decision, and reaction; Em-Garde goes a step further by entirely shifting instruction semantic parsing out-of-stream, leaving only proposal perception matching within the stream. The trade-off is its heavy reliance on whether proposals cover the true triggering evidence.
• vs StreamForest / FVStream: Both focus on managing and compressing growing video contexts and use assertive prompting for triggering. Em-Garde uses fixed short windows, ensuring throughput does not suffer as history grows, but consequently delegates long-range information to the proposer and downstream models instead of the matching module itself.
• vs StreamAgent / MMDuet-2: These works elevate complex triggering decisions via progressive planning or multi-round reinforcement learning, offering stronger expressiveness but at the cost of continuous reasoning. Em-Garde's RL only trains the query-time proposer, leaving the fast in-stream loop to lightweight cosine matching, which is better suited for tight real-time budgets.
• Insights for other tasks: Any system with "low-frequency condition updates and high-frequency sensor streams" can try pre-compiling conditions into detectable templates and running specialized matchers on the high-frequency path—provided that the templates are locally observable and matchmaking errors can be reliably calibrated.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulating frame-by-frame decisions into query-time propose and streaming-phase match is a clean task restructuring with broad system value.
• Experimental Thoroughness: ⭐⭐⭐⭐☆ Covers three proactive response benchmarks, two online comprehension benchmarks, throughput analysis, and multiple ablations, though lacking edge-side latency/power consumption and high-risk domain testing.
• Writing Quality: ⭐⭐⭐⭐☆ Motivations, module boundaries, and ablation relationships are clearly stated, though minor mathematical layout details and dataset naming descriptions remain slightly rough.
• Value: ⭐⭐⭐⭐⭐ Simultaneously improves triggering accuracy and constant-latency throughput, providing a highly practical architectural paradigm for long-running streaming video assistants.