EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/lingcco/EventMemAgent
Area: Video Understanding
Keywords: online video understanding, event memory, reservoir sampling, multi-granular perception, agentic reinforcement learning

TL;DR¶

EventMemAgent compresses video history into hierarchical event-centric memory and uses reinforcement learning to teach Qwen3-VL-8B when to retrieve, perform OCR, or detect objects, achieving an overall score of 60.75% on OVO-Bench with at most 32 frames in short-term memory.

Background & Motivation¶

Online video question answering cannot inspect an entire video before searching for an answer: a question arrives at a particular timestamp, and the system must use only earlier observations while remaining ready for subsequent frames. A sliding window forgets earlier events, while visual token compression can discard text or objects needed for detailed questions. Moving historical features to external storage extends the accessible time horizon, but retrieving many similar frames does not necessarily establish which action occurred or how states changed. Nor does it guarantee reliable evidence from small objects or unclear text.

Events offer a compression unit closer to the semantics of these questions. A prolonged painting sequence does not need another nearly identical summary every 30 seconds. Conversely, opening a sketchbook, dipping a brush in paint, and starting to paint should not be merged merely because they fall within one fixed temporal block. Compression nevertheless leaves information gaps: a caption may record that someone picked up a bottle without reading its label. Memory organization and active perception therefore need to be designed together. The former determines what remains retrievable; the latter determines whether a retrieved clue can be verified.

Rather than simply adding a retrieval database, the paper enables an agent to alternate between a limited set of visible frames and searchable event records, training its tool decisions through final-answer correctness. Core Idea: preserve video history as event records that support retrieval and renewed perception, use the short-term visual buffer for current details, and learn when to recall history or invoke specialized perception tools through reinforcement learning.

Method¶

Overall Architecture¶

The input video is sampled at 1 FPS. Event-centric short-term memory maintains current and recent events within a maximum of 32 frames, while structured long-term memory stores displaced history. When a question arrives, the agent reads short-term frames and selectively uses a multi-granular perception toolkit to retrieve historical events, read text, or locate objects before answering after multiple rounds of feedback.

Agentic RL separately optimizes these tool decisions during training; it is not an additional perception module executed at inference time. Solid edges below represent inference data flow, while dotted edges represent training supervision and policy optimization. All accessible visual evidence is constrained by the question timestamp.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Observed video stream"] --> STM["Event-centric short-term memory"]
    STM -->|Historical event eviction| LTM["Structured long-term memory"]
    STM --> Tools["Multi-granular perception toolkit"]
    LTM --> Tools
    Query["Question and timestamp"] --> Tools
    Tools -->|Tool feedback and further evidence gathering| Tools
    Tools --> Answer["Final answer"]
    Supervision["Training samples and answer correctness"] -.-> RL["Agentic RL"]
    RL -.->|Optimize tool decisions| Tools

Key Designs¶

1. Event-centric short-term memory: preserve event continuity before managing the frame budget

Short-term memory is not a simple queue of recent frames. It contains multiple events, with the last event continuing to receive incoming frames. For each new frame, the system computes a normalized grayscale histogram and compares it with the average histogram of frames already in the current event, using Pearson correlation to measure consistency. A correlation below the threshold of 0.2 closes the current event and starts another. This decision uses only observations already available, requiring neither future frames nor an offline segmentation model. However, grayscale appearance consistency is a proxy for semantic boundaries, not explicit recognition of action semantics.

New frames are appended while the total frame count across events remains below capacity. Once the buffer is full, there are two branches: if several events remain, the earliest event is evicted in FIFO order to accommodate new observations; if the entire buffer belongs to one continuing event, reservoir sampling is applied within that event instead of repeatedly splitting it. For its \(n\)-th processed frame and capacity \(K\), the probability of admitting the new frame is:

\[ P_{\mathrm{accept}}=\frac{K}{n},\qquad n>K. \]

An accepted frame replaces a randomly selected stored frame. Frames in a long event thus have equal inclusion opportunities, allowing the buffer to represent the event's full temporal span rather than only its most recent portion. The trade-off is that uniform representation does not guarantee preservation of rare critical frames: briefly visible text can still disappear. The boundary description mentions archiving, whereas the long-term memory section associates archiving with buffer eviction. The missing Appendix A promises the complete update rules, so the exact scheduling of these operations cannot be treated as a verified implementation detail.

2. Structured long-term memory: retain event content and changes between adjacent events

After a historical event enters long-term memory, its full frame sequence is replaced by four types of content: the first frame as a visual anchor, an MLLM-generated natural-language caption, a semantic embedding of that caption, and a change log describing state transitions between adjacent events. Start and end timestamps support later temporal retrieval. The caption describes what happened within an event, the change log describes what changed relative to the previous event, and the visual anchor supplies an image that can be passed to perception tools again.

This structure moves the long-term narrative outside the model context without assuming that a single summary can preserve every detail. Retrieval returns the complete event record rather than just a similarity score, providing both textual clues and an image for further inspection. The first frame, however, need not contain a critical state appearing later in the event. If a detail is absent from both the caption and the anchor, a later tool call cannot recover it from nothing. The bounded resources are the short-term frame budget and active context, not the storage size of the entire long-term database.

3. Multi-granular perception toolkit: verify evidence after retrieving a clue

Memory Search provides temporal and semantic retrieval. Temporal retrieval returns events overlapping a requested interval, while semantic retrieval searches by cosine similarity between the query embedding and event-caption embeddings. These support questions about a specified earlier interval and questions about whether a particular action occurred, respectively. Retrieved captions and change logs enter the context, allowing the agent to revise its query using newly discovered clues instead of relying on a single fixed retrieval pass.

Deepseek-OCR and Grounding DINO provide finer evidence through text extraction and object detection conditioned on target names. These tools can process either a selected short-term frame or a long-term event's visual anchor. Retrieval does not automatically trigger every tool: the agent chooses according to the unresolved question, using OCR for a label or detection for a particular object, then incorporates the observations before deciding whether to continue. Tool feedback therefore informs subsequent decisions rather than serving merely as a static attachment to the final answer.

4. Agentic RL: learn evidence-gathering strategies from answer feedback instead of hard-coded call sequences

The agent follows a ReAct-style interaction loop of reasoning, tool actions, and observations; an action may also terminate the interaction with an answer. During training, GRPO samples a group of trajectories from the old policy for each question and computes relative advantages by normalizing rewards with the group's mean and standard deviation, avoiding a separate critic. The paper explicitly assigns rewards solely by final-answer correctness: 1 for a correct answer and 0 otherwise. It reports neither per-tool supervision rewards nor an explicit penalty on the number of calls.

The order of memory search, OCR, and detection is consequently learned through answer-level credit assignment rather than fixed as a pipeline. The tool-use analysis reports that before training, more than 96% of samples fall into two extremes: no tool calls or repeated calls up to the turn limit. After training, historical retrieval and fine-grained perception are selected according to the task. This does not establish that RL directly optimizes latency; correctness rewards may still encourage longer outputs. The complete GRPO objective is corrupted in the cached text extraction, so this note explains the legible optimization mechanism without reconstructing the authors' exact objective.

A Worked Example¶

The following is an illustrative scenario constructed from the mechanism, not an additional experiment in the paper. Suppose the system has observed 64 consecutive frames from one painting event with a short-term capacity of 32. The probability that frame 64 enters the reservoir is \(32/64=0.5\). The buffer remains limited to 32 frames even as the event grows, while retaining opportunities to preserve earlier observations.

A subsequent scene change lowers the grayscale histogram correlation below 0.2 and starts a new event. Once archived, the painting event retains its first frame, caption, embedding, and change log rather than all 64 frames. If the user asks what was written on a bottle picked up earlier, the agent can retrieve the relevant event semantically and pass its first frame to OCR. This route fails if the bottle appeared only later in the event and its text was not recorded in the caption.

The example distinguishes finding a relevant event from retaining verifiable details. Hierarchical memory improves the former, while specialized tools help with the latter. Tools cannot overcome information already discarded by memory, a boundary that matters in deployment.

Loss & Training¶

The backbone is Qwen3-VL-8B-Instruct, trained directly with Agentic RL on 10K MovieChat samples labeled by VideoMarathon, leveraging the backbone's existing basic tool-use capabilities. Defaults are 1 FPS, a short-term capacity of 32, and an event threshold of 0.2. Frame indices and timestamps are added to short-term images. The main text reports training and evaluation on 8 A100 GPUs with 80GB each.

The separate efficiency experiment instead uses one 40GB A100 with vLLM; this is not the shared hardware configuration for every experiment. The text refers to Appendix B for deployment details and Appendix C for quantitative RL results, but the supplied cache ends with the references and does not contain those appendices. Learning rate, group size, maximum interaction turns, and untrained-agent accuracy are therefore not supplemented here.

Key Experimental Results¶

Main Results¶

Evaluation follows the online constraint, accessing only video before the question timestamp; training samples come from MovieChat rather than the two evaluation benchmarks below. The table reports OVO-Bench Overall from the paper's Table 1 and ALL for StreamingBench's real-time visual understanding portion from Table 2. Scores use a percentage scale, with higher values better. Comparisons across different backbones and sampling budgets are not strictly compute-matched experiments.

Method	Backbone Size	OVO-Bench Overall	StreamingBench ALL
Qwen3-VL	8B	55.81	70.20
StreamAgent	7B	49.40	74.28
StreamForest	7B	55.57	77.26
GPT-4o	Undisclosed	59.54	73.28
Gemini 1.5 Pro	Undisclosed	63.00	75.69
EventMemAgent	8B	60.75	77.00

EventMemAgent has a short-term visual budget of at most 32 frames. Qwen3-VL uses 64 frames on OVO-Bench and 0.2–1 FPS on StreamingBench, while both online baselines list 1 FPS. Gains over Qwen3-VL are 4.94 and 6.80 percentage points, respectively, but jointly reflect memory, tools, and additional training. EventMemAgent trails StreamForest by 0.26 percentage points on StreamingBench and does not exceed Gemini 1.5 Pro on OVO-Bench.

Ablation Study¶

The following summarizes the paper's Tables 3–4. The fixed-segmentation baseline creates a segment every 30 seconds while retaining the other agent components. Tool ablations are reported only on OVO-Bench, so missing entries are explicitly marked as not reported rather than treated as zero.

Config	OVO-Bench	Change vs. Full Model	StreamingBench	Change vs. Full Model
Full model	60.75	0.00	77.00	0.00
Fixed 30-second segment memory	58.27	-2.48	75.09	-1.91
Without object detection	57.80	-2.95	Not reported	Not reported
Without OCR	58.98	-1.77	Not reported	Not reported

These results support contributions from both event-level memory and specialized perception, but do not separately isolate event boundaries, reservoir sampling, visual anchors, or change logs. The main-results prose must also be distinguished from its table: Section 4.2 claims gains of 4.27 and 1.1 percentage points over existing open-source methods for real-time perception and forward active responding, respectively. In Table 1, however, the method's 68.29 and 55.92 compare with 65.52 and 55.73 for the strongest open-source comparator, Qwen3-VL, yielding differences of 2.77 and 0.19. This note retains the tabulated values and identifies the discrepancy rather than repeating those two prose claims.

Key Findings¶

Removing object detection reduces OVO-Bench overall performance by 2.95 percentage points, compared with 1.77 for OCR. The OCR subtask itself drops from 75.84 to 64.50, illustrating how an overall average can hide a tool's importance for a specific question type.
The efficiency experiment reports a median response latency of approximately 1.78 seconds on one 40GB A100 for videos extending to 30 minutes. MLLM generation accounts for 89.4% of latency and tool execution for 1.6%. This measures question-response latency, not the total cost of processing an unlimited video stream.
The paper describes StreamingBench as covering 12 tasks, but the real-time visual understanding portion in Table 2 lists 10 task abbreviations. This note reports that table's ALL score without extending it to complete coverage of every task setting.

Highlights & Insights¶

Event-centric buffering separates the decision to end a semantic segment from the decision to sample under a full frame budget. A long event is not mechanically split into repeated summaries solely because it reaches a frame-count limit.
Long-term memory supports both language retrieval and renewed visual inspection. Change logs retain narrative relationships, while visual anchors give specialized tools an image to process; these preserve different kinds of information.
Training changes whether the agent actively verifies evidence, not just how it generates answers. This can transfer to continuous-observation tasks, provided that historical memory retains the raw evidence required by the tools.

Limitations & Future Work¶

The authors note that StreamingBench emphasizes precise perception of current content, leaving long-term memory's potential incompletely demonstrated. Current results do not establish reasoning over extremely long histories.
Analysis in this note: grayscale histograms may confuse lighting or camera changes with event transitions and miss semantically different actions with similar appearances. Sampling at 1 FPS and uniform reservoir sampling can also miss brief critical states.
Analysis in this note: long-term storage grows with event count, while a single first frame and generated caption can create irreversible information bottlenecks. Longer-duration tests, storage-growth curves, and independent keyframe-preservation ablations would strengthen the conclusions.
Evidence limits: the supplied cache lacks Appendices A–E, and the corrupted GRPO formula was not reconstructed. The code URL comes from the paper's release statement; this task did not verify online repository availability.

vs StreamingVLM / VideoStreaming: Sliding windows or fixed-length token compression primarily manage visual information within context. This paper instead turns history into event records and lets an agent actively acquire further evidence, at the cost of external memory maintenance and multi-turn interaction.
vs M3-Agent / StreamAgent: According to the paper's related-work discussion, the former mainly limits online tools to memory retrieval, while the latter relies on prompt engineering. EventMemAgent expands perception tools and trains the decision policy, but does not establish through a full factorial ablation that gains originate from one particular change.
vs StreamForest: Both address online video memory, while this paper emphasizes event records and adaptive tool use. StreamingBench scores of 77.00 versus 77.26 show that active tool use is not necessarily advantageous for every online task.

Rating¶

Novelty: 4/5. Event-centric buffering, memory supporting renewed perception, and tool-policy training form a coherent system, although the individual mechanisms are not entirely new.
Experimental Thoroughness: 3/5. Two benchmarks, memory and tool ablations, and efficiency measurements are provided, but fine-grained memory ablations and verifiable quantitative RL results remain limited.
Writing Quality: 3/5. The main argument is clear, but some prose improvement claims disagree with the tables, and missing appendices limit implementation checks.
Value: 4/5. The framework is reusable for active online question answering within limited visual context while exposing the evidence bottleneck introduced by memory compression.